Computerized systems and methods for hosting and dynamically generating and providing customized media and media experiences

ABSTRACT

Disclosed are systems, servers and methods for providing a novel framework that enables the unique cataloging and organization of audio files, upon which audio rendering experiences can be created and provided to requesting users, whether the users are individuals or third party partners. The disclosed framework enables audio files to be stripped down, uniquely stored, and then stitched together in a novel manner that previously did not exist within the computing arts. The disclosed systems and methods, therefore, provide a novel platform where audio is not just provided to consumers, but audio experiences are compiled from various types of audio formats and types in a unique, dynamically determined manner for a listening user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of and priority to U.S. ProvisionalApplication No. 62/993,486, filed Mar. 23, 2020, entitled “Server,System And Method For Analyzing Files To Determine Overlay Suitability”,from U.S. Provisional Application No. 62/993,911, filed Mar. 24, 2020,entitled “System, Server And Method For Generating Files For ImprovedUser Experiences”, from U.S. Provisional Application No. 62/993,850,filed Mar. 24, 2020, entitled “Server, System and Method For ContentGeneration,” which are incorporated herein by reference in theirentirety.

This application is also a continuation-in-part of co-pending U.S.patent application Ser. No. 16/717,096, filed Dec. 17, 2019, entitled“Audio Content Production, Audio Sequencing, and Audio Blending Systemand Method,” which is a continuation application of U.S. patentapplication Ser. No. 15/336,627, filed Oct. 27, 2016, now U.S. Pat. No.10,509,622, entitled “Audio Content Production, Audio Sequencing, AndAudio Blending System And Method”, which claims the benefit of andpriority to U.S. Provisional Application No. 62/246,849, filed Oct. 27,2015, entitled “Audio Content Blending System And Method”, U.S.Provisional Application No. 62/254,072, filed Nov. 11, 2015, entitled“Production Content Creation Server, System and Method”, and U.S.Provisional Application No. 62/393,522, filed Sep. 12, 2016, entitled“Audio Content Sequencing”, which are incorporated herein by referencein their entirety.

This application includes material that is subject to copyrightprotection. The copyright owner has no objection to the facsimilereproduction by anyone of the patent disclosure, as it appears in thePatent and Trademark Office files or records, but otherwise reserves allcopyright rights whatsoever.

BACKGROUND

Digital and internet radio services have globally transformed thedelivery of audio content. The widespread use of digital formats, suchas, for example, compressed digital content, has provided broadcastersand other content providers with an almost infinite selection of contentfor a wide variety of uses. However, typical digital content experiencessuffer from playback gaps and other inartful characteristics whichundermine the listening experience and provide decision points forlisteners to abandon listening to the broadcast or other content.

Conversely, existing terrestrial radio stations and broadcast networkshave stagnated in their adoption of digital technologies, sometimesrelying on tools and techniques that are decades old and require humansto do tedious, repetitive, and menial tasks. In addition, maintainingthe quality of their product is a constant struggle, with even the mostwell-funded stations falling victim to human error and lesser-skilledlabor. These factors put them at a competitive disadvantage to newdigital delivery platforms.

SUMMARY

Some embodiments comprising a method for analyzing, by a computingdevice, an audio file, and determining attributes of the audio file, theattributes comprising information related to features of the audio file.Some embodiments further include determining, using the computingdevice, portions of the audio file that are eligible for mixing andportions that are ineligible; and generating, using the computingdevice, instructions for mixing audio data consistent with thedeterminations of the eligible and ineligible portions.

In some embodiments, the instructions are used to generate a stream ofaudio data that is output to a file or delivered to a network.

In some embodiments, the instructions are used to generate a stream ofaudio data which is sequenced to enable hitting the post at thebeginning of an ineligible portion.

In some embodiments, the audio data mixing includes a sequencedetermination that is based on a set of formulae, the formulaecomprising information for ordering audio data at predetermined times orintervals.

Some embodiments further comprise receiving input parameters from auser, the input parameters corresponding to at least some of thefeatures and characteristics of the audio file.

In some embodiments, the stream is a basis for a broadcast station.

Some embodiments further include at least one multidimensional databasethat comprises a plurality of data structures for specific types of theaudio features.

In some embodiments, the stream comprises song content and voice-overcontent.

In some embodiments, the audio file comprises third party content.

Currently, there does not exist a service, platform or provider that canbe configured to provide customized audio rendering experiences that arefully automated and seamlessly rendered versions of audio content.Beyond simply providing streams of music files that are retrofitted tounderstood behaviors or preferences of a user, conventional systems lackthe intelligence to provide a listening experience that includes varioustypes and formats of audio compiled based not only onsettings/parameters, but also the deep features discovered andunderstood from the audio included (or to be included) in a stream orother production or broadcast.

Some embodiments of the disclosed systems, servers and methods addressone or more of these shortcomings, among others, by providing animproved infrastructure that enables the unique cataloging andorganization of audio files and their subparts, upon which audiorendering experiences can be created and provided to requesting users,whether the users are individuals or third-party partners. As discussedherein, the disclosed framework enables audio files to be stripped down,uniquely stored, and then stitched together in a manner that previouslydid not exist within the computing arts. The disclosed systems andmethods, therefore, provide a novel platform where audio is not justprovided to consumers, but audio and listening experiences are generatedand compiled from various types of audio formats and types in a unique,dynamically determined manner for a listening user. As discussed herein,the listening experiences are provided in a manner that accounts fordata, instructions or some combination thereof, from users, contentproviders and the trained computer-models that are being implemented tocreate and provide the listening experiences.

According to some embodiments, a computer-implemented method isdisclosed, and includes steps including: receiving, by a computingdevice, over a network, a request to generate a stream of audio files,the request identifying at least an audio file; analyzing, by thecomputing device, the audio file, and determining attributes of theaudio file, the attributes comprising information related to featuresand characteristics of the audio file and acoustic content of the audiofile; generating, by the computing device, a query based on thedetermined attributes of the audio file; executing, by the computingdevice, in relation to at least one database associated with a platformprovided by the computing device, a search based on the generated query;identifying, by the computing device, a set of audio files based on thesearch, the set of audio files comprising at least two different typesof audio files, each audio file in the set of audio files comprising acontext that corresponds to the determined attributes of the audio file;determining, by the computing device, a data structure for playback ofthe audio file and the set of audio files, the determination comprising:determining a sequence for the playback, the sequence corresponding towhen one audio file begins playing respective to when another audio fileis rendered; and determining a mixdown between adjacently positionedaudio files within the determined sequence, the mixdown corresponding toan overlap in rendering of at least a portion of two adjacentlypositioned audio files; and facilitating, over the network, rendering ofthe playback.

In some embodiments, the analysis of the audio files that results in thedetermination of the attributes of the audio file is performed prior tothe operations for generating the playback. In some embodiments, theyare performed as audio files are identified. In some embodiments, theyare performed in response to a request from a user, third party orpartner. In some embodiments, prior analysis of like-content (with orwithout knowledge of the parameters that drove the audio's creation) canprovide an approximate set of attributes for that audio file.

In some embodiments, the sequence determination is based on a set offormulae (or formulas, used interchangeably), the formulae comprisinginformation for ordering audio files at predetermined times orintervals.

In some embodiments, a mixdown comprises: analyzing each audio fileassociated with the playback; and determining, for each audio file inthe playback, portions that are eligible for overlaying and portionsthat are ineligible for overlaying, wherein the mixdown determination isbased on the determined portions.

In some embodiments, the method further comprises: receiving inputparameters from a user, the input parameters corresponding to featuresand characteristics of the audio files included in the playback. In someembodiments, wherein the input parameters are utilized as part of theexecuted search. In some embodiments, the method further comprises:modifying the playback of at least one audio file based on the inputparameters.

In some embodiments, the method further comprises: storing the playbackdata structure at a network location, wherein a user can access theplayback audio files from the network location.

In some embodiments, the playback is a basis for a broadcast stationmade available over the network, wherein the audio files of the playbackare streamed over the network.

In some embodiments, the at least one database is a multidimensionaldatabase that comprises a plurality of data structures for specifictypes of audio features and characteristics. In some embodiments, thegenerated query is formatted as an n-dimensional query for searching themultidimensional database.

In some embodiments, the at least one database comprises portions ofaudio files, the portions corresponding to features and characteristicsof each audio file referenced in the at least one database.

In some embodiments, the playback comprises a set of audio files thatcomprise song content, and a set of audio files that comprise voice-overcontent. In some embodiments, the playback further comprises at leastone audio file comprising third party content. In some embodiments, theplayback further comprises at least one audio file comprising at leastone of interstitial information, upsell information and back-sellinformation. As discussed herein, audio types can include, but are notlimited to, songs, liners (e.g., voiceovers), interstitials, music beds,sound effects and content, and the like, or some combination thereof.

In some embodiments, the request comprises information describing theaudio file, wherein the information describing the audio filecorresponds to at least one of file type, name information, identifierand network location.

In some embodiments, the features and characteristics correspond to datarelated to, but not limited to, melodic features, tempo regions,amplitudes, beats per minute (BPM), fade ins/outs, features ofindividual stems (using source separation), dominant frequency ranges,structure, beat positions, onsets, harmonics, speakers/singer quantity,background noise, energy level, pitch, silence rates, duration, sonicgenre classification (multiple classifications with or without weights),loudness, key, meter, gender of vocals (male or female), arrangement(music with vocal or instrumental), mood (happiness and sadness),character (acousticness and electronicness), danceability, harmony(tonal or atonal), attitude (aggressiveness and chillness),environmentalness (music or environmental sounds), and environmentalsonic genre (multiple classifications with or without weights).

In some embodiments, a method is disclosed which includes the steps of:analyzing, by a computing device, an audio file, and determiningattributes of the audio file, the attributes comprising informationrelated to features of the audio file; determining, using the computingdevice, portions of the audio file that are eligible for overlaying andportions that are ineligible; generating, using the computing device, amixdown between the audio file and a second audio file, the mixdowncomprising ending the second audio file at a post; and providing, usingthe computing device, a rendering of the mixdown.

Some embodiments provide a non-transitory computer-readable storagemedium for carrying out the above-mentioned technical steps of theframework's functionality. The non-transitory computer-readable storagemedium has tangibly stored thereon, or tangibly encoded thereon,computer readable instructions that when executed by a device (e.g., aserver(s)) cause at least one processor to perform a method similar tothe method discussed above, and detailed in the instant disclosure.

In accordance with one or more embodiments, a system is provided thatcomprises one or more computing devices configured to providefunctionality in accordance with such embodiments. In accordance withone or more embodiments, functionality is embodied in steps of a methodperformed by at least one computing device. In accordance with someembodiments, program code (or program logic) executed by a processor(s)of a computing device to implement functionality in accordance with oneor more such embodiments is embodied in, by and/or on a non-transitorycomputer-readable medium.

DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features, and advantages of thedisclosure will be apparent from the following description ofembodiments as illustrated in the accompanying drawings, in whichreference characters refer to the same parts throughout the variousviews. The drawings are not necessarily to scale, emphasis instead beingplaced upon illustrating principles of the disclosure:

FIG. 1 is a schematic diagram illustrating an example of a networkwithin which the systems and methods disclosed herein could beimplemented according to some embodiments of the present disclosure;

FIG. 2 is a schematic diagram illustrating an example of a networkwithin which the systems and methods disclosed herein could beimplemented according to some embodiments of the present disclosure;

FIG. 3 depicts is a schematic diagram illustrating an example of clientdevice according to some embodiments of the present disclosure;

FIG. 4 is a block diagram illustrating components of an exemplary systemaccording to some embodiments of the present disclosure;

FIG. 5 is a flowchart detailing a non-limiting example embodimentaccording to some embodiments of the present disclosure;

FIG. 6 is a non-limiting example diagrammatic view of a networkedenvironment for processing audio files according to some embodiments ofthe present disclosure;

FIGS. 7A-7B are diagrams illustrating non-limiting example embodimentsof an audio file being processed according to some embodiments of thepresent disclosure;

FIG. 8 is a non-limiting example embodiment of an output file accordingto some embodiments of the present disclosure;

FIG. 9 is a flowchart illustrating a non-limiting example embodiment ofoperations for processing an audio file;

FIG. 10 is a schematic block diagram that provides a non-limitingexample embodiment a computing device in the networked environment ofFIG. 6 according to some embodiments of the present disclosure;

FIG. 11 is a flowchart detailing a non-limiting example embodimentaccording to some embodiments of the present disclosure;

FIG. 12 is a flowchart detailing a non-limiting example embodimentaccording to some embodiments of the present disclosure;

FIG. 13 is a flowchart illustrating a non-limiting example of theoperations of the mixdown agent executed in some embodiments of thenetworked environment of FIGS. 6 and 10 , according to some embodimentsof the present disclosure;

FIGS. 14A-14E are diagrams illustrating non-limiting examples of twoaudio files that are processed by the mixdown agent according to someembodiments of the present disclosure;

FIG. 15 is a diagram illustrating an example of an output file accordingto some embodiments of the present disclosure;

FIG. 16 is a diagram illustrating a non-limiting example embodiment ofthe operations of a content generator according to some embodiments ofthe present disclosure;

FIG. 17 is a diagram illustrating a non-limiting example embodiment ofaudio data processing via the content generator according to someembodiments of the present disclosure; and

FIG. 18 is flowchart detailing a non-limiting example embodimentaccording to some embodiments of the present disclosure.

DETAILED DESCRIPTION

The present disclosure will now be described more fully hereinafter withreference to the accompanying drawings, which form a part hereof, andwhich show, by way of non-limiting illustration, certain exampleembodiments. Subject matter may, however, be embodied in a variety ofdifferent forms and, therefore, covered or claimed subject matter isintended to be construed as not being limited to any example embodimentsset forth herein; example embodiments are provided merely to beillustrative. Likewise, a reasonably broad scope for claimed or coveredsubject matter is intended. Among other things, for example, subjectmatter may be embodied as methods, devices, components, or systems.Accordingly, embodiments may, for example, take the form of hardware,software, firmware or any combination thereof (other than software perse). The following detailed description is, therefore, not intended tobe taken in a limiting sense.

Throughout the specification and claims, terms may have nuanced meaningssuggested or implied in context beyond an explicitly stated meaning.Likewise, the phrase “in some embodiments” as used herein does notnecessarily refer to the same embodiment and the phrase “in anotherembodiment” as used herein does not necessarily refer to a differentembodiment. It is intended, for example, that claimed subject matterinclude combinations of example embodiments in whole or in part.

In general, terminology may be understood at least in part from usage incontext. For example, terms, such as “and”, “or”, or “and/or,” as usedherein may include a variety of meanings that may depend at least inpart upon the context in which such terms are used. Typically, “or” ifused to associate a list, such as A, B or C, is intended to mean A, B,and C, here used in the inclusive sense, as well as A, B or C, here usedin the exclusive sense. In addition, the term “one or more” as usedherein, depending at least in part upon context, may be used to describeany feature, structure, or characteristic in a singular sense or may beused to describe combinations of features, structures or characteristicsin a plural sense. Similarly, terms, such as “a,” “an,” or “the,” again,may be understood to convey a singular usage or to convey a pluralusage, depending at least in part upon context. In addition, the term“based on” may be understood as not necessarily intended to convey anexclusive set of factors and may, instead, allow for the existence ofadditional factors not necessarily expressly described, again, dependingat least in part on context.

The present disclosure is described below with reference to blockdiagrams and operational illustrations of methods and devices. It isunderstood that each block of the block diagrams or operationalillustrations, and combinations of blocks in the block diagrams oroperational illustrations, can be implemented by means of analog ordigital hardware and computer program instructions. These computerprogram instructions can be provided to a processor of a general purposecomputer to alter its function as detailed herein, a special purposecomputer, ASIC, or other programmable data processing apparatus, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, implement thefunctions/acts specified in the block diagrams or operational block orblocks. In some alternate implementations, the functions/acts noted inthe blocks can occur out of the order noted in the operationalillustrations. For example, two blocks shown in succession can in factbe executed substantially concurrently or the blocks can sometimes beexecuted in the reverse order, depending upon the functionality/actsinvolved.

For the purposes of this disclosure, a non-transitory computer readablemedium (or computer-readable storage medium/media) stores computer data,which data can include computer program code (or computer-executableinstructions) that is executable by a computer, in machine readableform. By way of example, and not limitation, a computer readable mediummay comprise computer readable storage media, for tangible or fixedstorage of data, or communication media for transient interpretation ofcode-containing signals. Computer readable storage media, as usedherein, refers to physical or tangible storage (as opposed to signals)and includes without limitation volatile and non-volatile, removable andnon-removable media implemented in any method or technology for thetangible storage of information such as computer-readable instructions,data structures, program modules or other data. Computer readablestorage media includes, but is not limited to, RAM, ROM, EPROM, EEPROM,flash memory or other solid state memory technology, CD-ROM, DVD, orother optical storage, cloud storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices, or any otherphysical or material medium which can be used to tangibly store thedesired information or data or instructions and which can be accessed bya computer or processor.

For the purposes of this disclosure the term “server” should beunderstood to refer to a service point which provides processing,database, and communication facilities. By way of example, and notlimitation, the term “server” can refer to a single, physical processorwith associated communications and data storage and database facilities,or it can refer to a networked or clustered complex of processors andassociated network and storage devices, as well as operating softwareand one or more database systems and application software that supportthe services provided by the server. Cloud servers are examples.

For the purposes of this disclosure, a “network” should be understood torefer to a network that may couple devices so that communications may beexchanged, such as between a server and a client device or other typesof devices, including between wireless devices coupled via a wirelessnetwork, for example. A network may also include mass storage, such asnetwork attached storage (NAS), a storage area network (SAN), a contentdelivery network (CDN) or other forms of computer or machine readablemedia, for example. A network may include the Internet, one or morelocal area networks (LANs), one or more wide area networks (WANs),wire-line type connections, wireless type connections, cellular or anycombination thereof. Likewise, sub-networks, which may employ differingarchitectures or may be compliant or compatible with differingprotocols, may interoperate within a larger network.

For purposes of this disclosure, a “wireless network” should beunderstood to couple client devices with a network. A wireless networkmay employ stand-alone ad-hoc networks, mesh networks, Wireless LAN(WLAN) networks, cellular networks, or the like. A wireless network mayfurther employ a plurality of network access technologies, includingWi-Fi, Long Term Evolution (LTE), WLAN, Wireless Router (WR) mesh, or2nd, 3rd, 4^(th) or 5^(th) generation (2G, 3G, 4G or 5G) cellulartechnology, mobile edge computing (MEC), Bluetooth™, 802.11b/g/n, or thelike. Network access technologies may enable wide area coverage fordevices, such as client devices with varying degrees of mobility, forexample.

In short, a wireless network may include any type of wirelesscommunication mechanism by which signals may be communicated betweendevices, such as a client device or a computing device, between orwithin a network, or the like.

A computing device may be capable of sending or receiving signals, suchas via a wired or wireless network, or may be capable of processing orstoring signals, such as in memory as physical memory states, and may,therefore, operate as a server. Thus, devices capable of operating as aserver may include, as examples, dedicated rack-mounted servers, desktopcomputers, laptop computers, set top boxes, integrated devices combiningvarious features, such as two or more features of the foregoing devices,or the like.

For purposes of this disclosure, a client (or consumer or user) devicemay include a computing device capable of sending or receiving signals,such as via a wired or a wireless network. A client device may, forexample, include a desktop computer or a portable device, such as acellular telephone, a smart phone, a display pager, a radio frequency(RF) device, an infrared (IR) device an Near Field Communication (NFC)device, a Personal Digital Assistant (PDA), a handheld computer, atablet computer, a phablet, a laptop computer, a set top box, a wearablecomputer, smart watch, an integrated or distributed device combiningvarious features, such as features of the forgoing devices, or the like.

A client device may vary in terms of capabilities or features. Claimedsubject matter is intended to cover a wide range of potentialvariations, such as a web-enabled client device or previously mentioneddevices may include a high-resolution screen (HD or 4K for example), oneor more physical or virtual keyboards, mass storage, one or moreaccelerometers, one or more gyroscopes, global positioning system (GPS)or other location-identifying type capability, or a display with a highdegree of functionality, such as a touch-sensitive color 2D or 3Ddisplay, for example.

Certain embodiments will now be described in greater detail withreference to the figures. In general, with reference to FIG. 1 , asystem 100 in accordance with some embodiments of the present disclosureis shown. FIG. 1 shows components of a general environment in which thesystems and methods discussed herein may be practiced. Not all thecomponents may be required to practice the disclosure, and variations inthe arrangement and type of the components may be made without departingfrom the spirit or scope of the disclosure. As shown, system 100 of FIG.1 includes local area networks (“LANs”)/wide area networks(“WANs”)—network 105, wireless network 110, mobile devices (clientdevices) 102-104 and client device 101. FIG. 1 additionally includes avariety of servers, such as content server 106, application (or “App”)server 108 and third party server 130.

Some embodiments of mobile devices 102-104 may include virtually anyportable computing device capable of receiving and sending a messageover a network, such as network 105, wireless network 110, or the like.Mobile devices 102-104 may also be described generally as client devicesthat are configured to be portable. Thus, mobile devices 102-104 mayinclude virtually any portable computing device capable of connecting toanother computing device and receiving information, as discussed above.

Mobile devices 102-104 also may include at least one client applicationthat is configured to receive content from another computing device. Insome embodiments, mobile devices 102-104 may also communicate withnon-mobile client devices, such as client device 101, or the like. Insome embodiments, such communications may include sending and/orreceiving messages, creating and uploading documents, searching for,viewing and/or sharing memes, photographs, digital images, audio clips,video clips, or any of a variety of other forms of communications.

Client devices 101-104 may be capable of sending or receiving signals,such as via a wired or wireless network, or may be capable of processingor storing signals, such as in memory as physical memory states, andmay, therefore, operate as a server or other appropriately configuredcomputer.

In some embodiments, wireless network 110 is configured to couple mobiledevices 102-104 and its components with network 105. Wireless network110 may include any of a variety of wireless sub-networks that mayfurther overlay stand-alone ad-hoc networks, and the like, to provide aninfrastructure-oriented connection for mobile devices 102-104.

In some embodiments, network 105 is configured to couple content server106, application server 108, or the like, with other computing devices,including, client device 101, and through wireless network 110 to mobiledevices 102-104. Network 105 is enabled to employ any form of computerreadable media or network for communicating information from oneelectronic device to another.

In some embodiments, the content server 106 may include a device thatincludes a configuration to provide any type or form of content via anetwork to another device. Devices that may operate as content server106 include personal computers, desktop computers, multiprocessorsystems, microprocessor-based or programmable consumer electronics,network PCs, servers, and the like. In some embodiments, content server106 can further provide a variety of services that include, but are notlimited to, email services, instant messaging (IM) services, streamingand/or downloading media services, search services, photo services, webservices, social networking services, news services, third-partyservices, audio services, video services, SMS services, MMS services,FTP services, voice over IP (VOIP) services, or the like.

Third party server 130 can comprise a server that stores advertisementsfor presentation/rendering to users. “Ad serving” refers to methods usedto stream online, audio advertisement files to users over a network, asusers are streaming media content, and the like. Various monetizationtechniques or models may be used in connection with sponsoredadvertising, including advertising associated with user data. Suchsponsored advertising includes, but is not limited to, monetizationtechniques including sponsored advertising, non-sponsored advertising,guaranteed and non-guaranteed delivery advertising, adnetworks/exchanges, ad targeting, ad serving and ad analytics. Suchsystems can incorporate near instantaneous auctions of ad placement andinsertion into media streams, (in some cases in less than 500milliseconds) with higher quality audio ad placement opportunitiesresulting in higher revenues per ad. That is, advertisers will payhigher advertising rates when they believe their ads are being placed inor along with highly relevant content that is being presented to users.Reductions in the time needed to quantify a high quality ad placementoffers ad platforms competitive advantages. Thus, higher speeds and morerelevant context detection improve these technological fields.

Another approach includes profile-type ad targeting. In this approach,user profiles specific to a user may be generated to model userbehavior, for example, by tracking a user's path through a service, website or network of sites, and compiling a profile based at least in parton pages or advertisements ultimately delivered. A correlation may beidentified, such as for user purchases, for example. An identifiedcorrelation may be used to target potential purchasers by targetingcontent or advertisements to particular users. During providing ofadvertisements, a presentation system may collect descriptive contentabout types of advertisements presented to users. A broad range ofdescriptive content may be gathered, including content specific to anadvertising presentation system. Advertising analytics gathered may betransmitted to locations remote to an advertising presentation systemfor storage or for further evaluation. Where advertising analyticstransmittal is not immediately available, gathered advertising analyticsmay be stored by an advertising presentation system until transmittal ofthose advertising analytics becomes available.

In some embodiments, users are able to access services provided byservers 106, 108 and 130. This may include in a non-limiting example,authentication servers, search servers, email servers, social networkingservices servers, SMS servers, IM servers, MMS servers, exchangeservers, photo-sharing services servers, and travel services servers,via the network 105 using their various devices 101-104.

In some embodiments, application server 108, for example, can storevarious types of applications and application related informationincluding application data, media file programming information, and userprofile information. It should also be understood that content server106 can also store various types of data related to the content andservices provided by content server 106 in an associated contentdatabase 107, as discussed in more detail below. Embodiments exist wherethe network 105 is also coupled with/connected to a Trusted SearchServer (TSS) which can be utilized to render content in accordance withthe embodiments discussed herein. Embodiments exist where the TSSfunctionality can be embodied within servers 106, 108 and/or 130.

In some embodiments, servers 106, 108 and/or 130 can be embodied as acloud server or configured for hosting cloud services, as discussedherein.

Moreover, although FIG. 1 illustrates servers 106, 108 and 130 as singlecomputing devices, respectively, the disclosure is not so limited. Forexample, one or more functions of servers 106, 108 and/or 130 may bedistributed across one or more distinct computing devices. Moreover, inone embodiment, servers 106, 108 and/or 130 may be integrated into asingle computing device, without departing from the scope of the presentdisclosure.

Additionally, while the illustrated embodiment in FIG. 1 depicts onlyservers 106, 108 and 130, it should not be construed as limiting, as anytype and number of servers can be included therein. Further, whilecontent database 107 is depicted as a single database, it should not beconstrued as limiting, as any type and number of databases can beincluded therein, as discussed in more detail below.

Turning to FIG. 2 , computer system 210 is depicted and is anon-limiting example embodiment of system 100 discussed above inrelation to FIG. 1 .

FIG. 2 illustrates a computer system 210 enabling or operating anembodiment of system 100 of FIG. 1 , as discussed below. In someembodiments, computer system 210 can include and/or operate and/orprocess computer-executable code of one or more of the above-mentionedprogram logic, software modules, and/or systems. Further, in someembodiments, the computer system 210 can operate and/or displayinformation within one or more graphical user interfaces. In someembodiments, the computer system 210 can comprise a cloud server and/orcan be coupled to one or more cloud-based server systems.

In some embodiments, the system 210 can comprise at least one computingdevice 230 including at least one processor 232. In some embodiments,the at least one processor 232 can include a processor residing in, orcoupled to, one or more server platforms. In some embodiments, thesystem 210 can include a network interface 235 a and an applicationinterface 235 b coupled to the least one processor 232 capable ofprocessing at least one operating system 234. Further, in someembodiments, the interfaces 235 a, 235 b coupled to at least oneprocessor 232 can be configured to process one or more of the softwaremodules 238 (e.g., such as enterprise applications). In someembodiments, the software modules 238 can include server-based software,and can operate to host at least one user account and/or at least oneclient account, and operating to transfer data between one or more ofthese accounts using the at least one processor 232.

With the above embodiments in mind, it should be understood that someembodiments can employ various computer-implemented operations involvingdata stored in computer systems. Moreover, the above-described databasesand models described throughout can store analytical models and otherdata on computer-readable storage media within the system 210 and oncomputer-readable storage media coupled to the system 210. In addition,the above-described applications of the system can be stored onnon-transitory computer-readable storage media within the system 210 andon computer-readable storage media coupled to the system 210.

In some embodiments, the system 210 can comprise at least onenon-transitory computer readable medium 236 coupled to at least one datasource 237 a, and/or at least one data storage device 237 b, and/or atleast one input/output device 237 c. In some embodiments, the disclosedsystems and methods can be embodied as computer readable code on acomputer readable medium 236. In some embodiments, the computer readablemedium 236 can be any data storage device that can store data, which canthereafter be read by a computer system (such as the system 210). Insome embodiments, the computer readable medium 236 can be any physicalor material medium that can be used to tangibly store the desiredinformation or data or instructions and which can be accessed by acomputer or processor 232. In some embodiments, at least one of thesoftware modules 238 can be configured within the system to output datato at least one user 231 via at least one graphical user interfacerendered on at least one digital display.

In some embodiments, the non-transitory computer readable medium 236 canbe distributed over a conventional computer network via the networkinterface 235 a where the system embodied by the computer readable codecan be stored and executed in a distributed fashion. For example, insome embodiments, one or more components of the system 210 can becoupled to send and/or receive data through a local area network (“LAN”)239 a and/or an internet coupled network 239 b (e.g., such as a wirelessinternet). In some further embodiments, the networks 239 a, 239 b caninclude wide area networks (“WAN”), direct connections (e.g., through auniversal serial bus port), or other forms of computer-readable media236, or any combination thereof.

In some embodiments, components of the networks 239 a, 239 b can includeany number of user devices such as personal computers including forexample desktop computers, and/or laptop computers, or any fixed,generally non-mobile internet appliances coupled through the LAN 239 a.For example, some embodiments include personal computers 240 a coupledthrough the LAN 239 a that can be configured for any type of userincluding an administrator. Other embodiments can include personalcomputers coupled through network 239 b. In some further embodiments,one or more components of the system 210 can be coupled to send orreceive data through an internet network (e.g., such as network 239 b).For example, some embodiments include at least one user 231 coupledwirelessly and accessing one or more software modules of the systemincluding at least one enterprise application 238 via an input andoutput (“I/O”) device 237 c. In some other embodiments, the system 210can enable at least one user 231 to be coupled to access enterpriseapplications 238 via an I/O device 237 c through LAN 239 a. In someembodiments, the user 231 can comprise a user 231 a coupled to thesystem 210 using a desktop computer, and/or laptop computers, or anyfixed, generally non-mobile internet appliances coupled through theinternet 239 b. In some embodiments, the user 231 can comprise a mobileuser 231 b coupled to the system 210. In some embodiments, the user 231b can use any mobile computing device 231 c to wirelessly coupled to thesystem 210, including, but not limited to, personal digital assistants,and/or cellular phones, mobile phones, or smart phones, and/or pagers,and/or digital tablets, and/or fixed or mobile internet appliances.

FIG. 3 is a schematic diagram illustrating a client device showing anexample embodiment of a client device that may be used within thepresent disclosure. Client device 300 may include many more or fewercomponents than those shown in FIG. 3 . However, the components shownare sufficient to disclose an illustrative embodiment for implementingthe present disclosure. Client device 300 may represent, for example,client devices discussed above in relation to FIGS. 1-2 .

As shown in FIG. 3 , in some embodiments, client device 300 includes aprocessing unit (CPU) 322 in communication with a mass memory 330 via abus 324. In some embodiments, client device 300 also includes a powersupply 326, one or more network interfaces 350, an audio interface 352,a display 354, a keypad 356, an illuminator 358, an input/outputinterface 360, a haptic interface 362, an optional global positioningsystems (GPS) receiver 364 and a camera(s) or other optical, thermal orelectromagnetic sensors 366. Device 300 can include one camera/sensor366, or a plurality of cameras/sensors 366, as understood by those ofskill in the art. Power supply 326 provides power to the client device300.

Client device 300 may optionally communicate with a conventional basestation (not shown), or directly with another computing device. Networkinterface 350 is sometimes known as a transceiver, transceiving device,or network interface card (NIC).

In some embodiments, audio interface 352 is arranged to produce andreceive audio signals such as the sound of a human voice. Display 354may be a liquid crystal display (LCD), gas plasma, light emitting diode(LED), or any other type of display used with a computing device.Display 354 may also include a touch sensitive screen arranged toreceive input from an object such as a stylus or a digit from a humanhand.

Keypad 356 may comprise any input device arranged to receive input froma user. Illuminator 358 may provide a status indication and/or providelight.

In some embodiments, client device 300 also comprises input/outputinterface 360 for communicating with external. Input/output interface360 can utilize one or more communication technologies, such as USB,NFC, infrared, Bluetooth™, or the like. In some embodiments, hapticinterface 362 is arranged to provide tactile feedback to a user of theclient device.

Optional GPS transceiver 364 can determine the physical coordinates ofclient device 300 on the surface of the Earth, which typically outputs alocation as latitude and longitude values. GPS transceiver 364 can alsoemploy other geo-positioning mechanisms, including, but not limited to,triangulation, assisted GPS (AGPS), E-OTD, CI, SAI, ETA, BSS or thelike, to further determine the physical location of Client device 300 onthe surface of the Earth. In some embodiments, however, the clientdevice 300 may through other components, provide other information thatmay be employed to determine a physical location of the device,including for example, a MAC address, Internet Protocol (IP) address, orthe like.

In some embodiments, mass memory 330 includes a RAM 332, a ROM 334, andother storage means. Mass memory 330 illustrates another example ofcomputer storage media for storage of information such as computerreadable instructions, data structures, program modules or other data.Mass memory 330 stores a basic input/output system (“BIOS”) 340 forcontrolling low-level operation of client device 300. The mass memoryalso stores an operating system 341 for controlling the operation ofclient device 300.

In some embodiments, memory 330 further includes one or more datastores, which can be utilized by client device 300 to store, among otherthings, applications 342 and/or other information or data. For example,data stores may be employed to store information that describes variouscapabilities of client device 300. The information may then be providedto another device based on any of a variety of events, including beingsent as part of a header (e.g., index file of the HLS stream) during acommunication, sent upon request, or the like. At least a portion of thecapability information may also be stored on a disk drive or otherstorage medium (not shown) within client device 300.

In some embodiments, applications 342 may include computer executableinstructions which, when executed by client device 300, transmit,receive, and/or otherwise process audio, video, images, and enabletelecommunication with a server and/or another user of another clientdevice. In some embodiments, applications 342 may further include searchclient 345 that is configured to send, to receive, and/or to otherwiseprocess a search query and/or search result.

Having described the components of the general architecture employedwithin some embodiments, the components' general operation with respectto some embodiments will now be described below.

FIG. 4 is a block diagram illustrating the components of someembodiments. FIG. 4 includes media engine 400, network 415 and database420. The media engine 400 can be a special purpose machine or processorand could be hosted by a cloud server (e.g., cloud web servicesserver(s)), application server, content server, web server, searchserver, content provider, third party server, user's computing device,and the like, or any combination thereof.

According to some embodiments, media engine 400 can be embodied as astand-alone application that executes on a server and/or user device(e.g., on a cloud server and/or on-prem on a user device or localstorage). In some embodiments, the media engine 400 can function as anapplication installed on a device. In some embodiments, such applicationcan be a web-based application accessed by a device over a network. Insome embodiments, the application can be a software development kit(SDK) or an application program interface (API), and the like.

The database 420 can be any type of database or memory and can beassociated with a content server on a network (e.g., cloud server,content server, a search server or application server) or a user'sdevice (e.g., client devices discussed above in FIGS. 1-3 ). Similarlyto database 107 of FIG. 1 , database 420 can be any type and number ofdatabases, as discussed in more detail below.

Database 420 comprises a dataset of data and metadata associated withlocal and/or network information related to users, services,applications, content and the like. Such information can be stored andindexed in the database 420 independently and/or as a linked orassociated dataset. As discussed above and in more detail below, itshould be understood that the data (and metadata) in the database 420can be any type of information and type, whether known or to be known,without departing from the scope of the present disclosure.

In some embodiments, database 420 can be configured as amultidimensional database that houses separate structures for handlingdifferent types of content files, content portions, and content portionconfigurations (e.g., feature vector data versus content data versuskey-values, versus tags and tokens, and the like). It should beunderstood by those of skill in the art that any type of known or to beknown type, format or version of multidimensional databases and/orvector similarity search engines (e.g., Annoy or Faiss) can be utilizedwithout departing from the scope of the instant disclosure.

According to some embodiments, database 420 can store data for users,e.g., user data. According to some embodiments, the stored user data caninclude, but is not limited to, information associated with a user'sprofile, user interests, user behavioral information, user attributes,user preferences or settings, user demographic information, userlocation information, user biographic information, and the like, or somecombination thereof.

In some embodiments, the user data can also include, for purposesproviding, displaying, creating, streaming, recommending, renderingand/or delivering media, user device information, including, but notlimited to, device identifying information, device capabilityinformation, device display attributes (e.g., screen size, resolution,version, and the like), voice/data carrier information, InternetProtocol (IP) address, applications installed or capable of beinginstalled or executed on such device, and/or any, or some combinationthereof.

It should be understood that the data (and metadata) in the database 420can be any type of information related to a user, content, a device, anapplication, a service provider, a content provider, whether known or tobe known, without departing from the scope of the present disclosure.

According to some embodiments, database 420 can store data and metadataassociated with media content from an assortment of media and/or serviceproviders and/or platforms. For example, the information can be relatedto, but not limited to, content type of the media file, a categoryassociated with the media, information associated with the audio qualityand attributes (for example), information associated with the provideror hosting entity of the media, and any other type of known or to beknown attribute or feature associated with a media file. Additionally,the media information in database 420 for each media file can comprise,but is not limited to, attributes including, but not limited to,popularity of the media, quality of the media, recency of the media(when it was published, shared, edited and the like), and the like. Suchfactors can be derived from information provided by the user, a serviceprovider, by the content/service providers providing media content, orby other third party services (e.g., Facebook®, Twitter® and the like),or some combination thereof.

According to some embodiments, database 420 can store data and metadataassociated with media files, including, but not limited to, audio files,video files, text files, multi-media files, and the like, or somecombination thereof. The data/metadata can further include, but is notlimited to, information related to users, products, applications,services, media providers, service providers, and the like, or somecombination thereof. It should be understood that the data (andmetadata) in the database 420 can be any type of information related toa user, media file, an application, a service provider, a contentprovider, whether known or to be known, without departing from the scopeof the present disclosure.

While the focus on this disclosure will refer to audio files, it shouldnot be construed as limiting, as any other type of media file, whetherknown or to be known, can be implemented without departing from thescope of the instant application. For example, audio files, as discussedherein, can be music files (e.g., songs), voice-overs (e.g., short clipsof commentary), advertisements (e.g., 30 second audio ads from thirdparties), instrumentals (e.g., music beds), sound effects, and the like.

According to some embodiments, the media data/metadata can be stored indatabase 420 as an n-dimensional vector (or feature vector)representation for each media, where the information associated with themedia can be translated as a node on the n-dimensional vector. Database420 can store and index media information in database 420 as linked setof media data and metadata, where the data and metadata relationship canbe stored as the n-dimensional vector. Such storage can be realizedthrough any known or to be known vector or array storage, including butnot limited to, a hash tree, queue, stack, VList, or any other type ofknown or to be known dynamic memory allocation technique or technology.While the storage discussion herein focuses on vector analysis, thestored information in database 420 can be analyzed, stored and indexedaccording to any known or to be known computational analysis techniqueor algorithm, such as, but not limited to, cluster analysis, datamining, vector search engines, Bayesian network analysis, Hidden Markovmodels, artificial neural network analysis (ANN), convolutional neuralnetworks (CNN), recurrent neural networks (RNNs), logical model and/ortree analysis, and the like. Additionally, the user data stored indatabase 420 can be stored in a similar manner.

As discussed above, with reference to FIGS. 1-2 , the network 415 can beany type of network such as, but not limited to, a wireless network, alocal area network (LAN), wide area network (WAN), the Internet, or acombination thereof. The network 415 facilitates connectivity of themedia engine 400, and the database of stored resources 420. Indeed, asillustrated in FIG. 4 , the media engine 400 and database 420 can bedirectly connected by any known or to be known method of connectingand/or enabling communication between such devices and resources.

The principal processor, server, or combination of devices thatcomprises hardware programmed in accordance with the special purposefunctions herein is referred to for convenience as media engine 400, andincludes audio processing module 402, storage module 404, playbackmodule 406 and generator module 408. It should be understood that theengine(s) and modules discussed herein are non-exhaustive, as additionalor fewer engines and/or modules (or sub-modules) may be applicable tothe embodiments of the systems and methods discussed. The operations,configurations and functionalities of each module, and their role withinembodiments of the present disclosure will be discussed below.

Turning to FIG. 5 , Process 500 provides non-limiting exampleembodiments for processing a media file in order for generation of anoutput experience (e.g., storage into a folder, a playlist, radiostation, and the like, as discussed below). According to someembodiments, Steps 502-504 of Process 500 are performed by audioprocessing module 402 of media engine 400; Step 506 is performed bystorage module 404; Step 508 is performed by playback module 406; andStep 510 is performed by generator module 408.

As mentioned above, for purposes of this disclosure, the media filebeing processed will be discussed in relation to an audio file, however,it should not be construed as limiting, as any type of media fileformat, whether known or to be known, can be utilized, analyzed andprocessed according to the disclosed system and methods discussed hereinwithout departing from the scope of the instant disclosure.

Process 500 begins with Step 502 where an audio file is identified. Insome embodiments, a set or plurality of audio files can be identified;however, for purposes of discussion in relation to Process 500, a singleaudio file will be discussed as being identified for clarification ofoperation purposes, as one of ordinary skill in the art would readilyrecognize that any number of audio files can be processed accordingly,either sequentially or as part of a batch operation.

In some embodiments, the audio file can be a licensed audio file, a usergenerated content (UGC) file, a network located audio file, and thelike. In some embodiments, Step 502 can involve requesting an audio filefrom a third party provider, where it can be retrieved and/or providedaccordingly. In some embodiments, Step 502 can involve downloading thefile from a network location (e.g., hosted to a file transfer (FTP)site, from where it is retrieved). In some embodiments, Step 502 caninvolve retrieving the file from an associated database (e.g., a localor network accessible datastore from which a collection of audio filesis maintained).

In some embodiments, the audio file has associated therewith informationindicating, but not limited to, a type of audio file (e.g., music, voiceover, and the like), and an audio identifier (ID) (which can be aninternal ID or an ID associated with the provider of the file). In someembodiments, this information can further indicate a source of the audiofile, length, size, descriptive tags, relationships to other content,relationships to organizing entities (e.g., the voice, artist, album,record label, advertiser, content channel, and the like), associatedvisual images, and the like or some combination thereof. Thisinformation can be leveraged to store and provide the audio, asdiscussed in more detail below.

In Step 504, the audio file is analyzed. According to some embodiments,the analysis of the audio file results in the determination, detection,retrieval, extraction or otherwise identification of attributes,characteristics, features, deep descriptors, and the like, or somecombination thereof, of the audio file.

In some embodiments, the analysis performed in Step 504 of the audiofile from Step 502 can involve analysis and identification ofdata/metadata by any known or to be known audio (or media) file analysistechnique, algorithm, classifier or mechanism, including, but notlimited to, ANNs, CNNs, RNNs, generative adversarial networks (GANNs),source separation with recursive stem analysis, audio segmentation andlabeling, predominant melody extraction, harmonic chord progressionextraction, onset detection, beat detection, downbeat detection, rubatodetection, neural BPM identification (which leverages multiple DSP, RNN,and CNN techniques), amplitude analysis and directional profiling,computer vision, Bayesian network analysis, Hidden Markov Models, datamining, feature vector analysis, logical model and/or tree analysis, andthe like.

In some embodiments, such analysis, as discussed herein, and whendiscussed below in relation to similar analysis, identification anddetermination steps, can involve using any known or to be known deeplearning architecture or algorithm, such as, but not limited to, deepneural networks, ANNs, CNNs, deep belief networks and the like.

According to some embodiments, engine 400 employs CNNs (however, itshould not be construed to limit the present disclosure to only theusage of CNNs, as any known or to be known deep learning architecture oralgorithm is applicable to the disclosed systems and methods discussedherein). CNNs, as discussed herein, can operate on a CPU or on anassociated graphics processing unit (GPU) for enhanced performance. CNNsconsist of multiple layers which can include: the convolutional layer,ReLU (rectified linear unit) layer, pooling layer, dropout layer andloss layer, as understood by those of skill in the art. When used foraudio recognition, CNNs produce multiple tiers of deep featurecollections by analyzing small portions of an input audio file, whichenables the identification and viewing of portions and/or an entirety ofan audio file, as well as its changes over time.

For purposes of this disclosure, such features/descriptors can include,but are not limited to, audio characteristics of the audio files (or“slices” or portions of the file) characterized (or categorized andlabeled) by acoustic features, melodic features, type features, harmonicfeatures, rhythm features, and the like, or some combination thereof.The results of these collections are then tiled so that they overlap toobtain a better representation of the original audio; which is repeatedfor every CNN layer. CNNs may include local or global pooling layers,which combine the outputs of feature clusters. One advantage of CNNs isthe use of shared weight in convolutional layers; that is, the samefilter (weights) is used for each audio portion in each layer, therebyreducing required memory size and improving performance. Compared toother classification algorithms, CNNs use relatively littlepre-processing which avoids the dependence on prior-knowledge and theexistence of difficult to design handcrafted features.

According to some embodiments, engine 400 can perform the analysis ofStep 504 and determine i) dominance drive values; ii) confidence drivenvalues and/or iii) matrix values. These values can be compared against athreshold to ensure a minimum amount of dominance, confidence and/or“order” (from the matrix values) are provided or observed.

In some embodiments, dominance driven values represent the dominantfeature from a pool of two or more features. The pool can be for anaudio file, or a set of audio files. Two features are usuallyrepresented by a floating point number that orients the observationbetween the two features (e.g., tonalness). Three or more features arerepresented by an integer and a companion confidence measure (e.g.,genre).

In some embodiments, confidence driven values represent how confidentlythe feature was observed. This is not to be confused with a bipolarmeasure; the low-confidence observation does not necessarily representthe corollary of the feature. For example, low happiness does not meansadness was detected. In order to evaluate how sad the content is,engine 400 may need to use a sadness measure. In some embodiments, thesefeatures are identified independently and may, in some cases, conflictwith each other.

In some embodiments, matrix values are an ordered array of values thatconsist of an ID and a confidence value. In some embodiments, matrixvalues can primarily be used to convey the complex output of amulti-value analysis model (e.g., Music Genre).

In some embodiments, engine 400 can be configured to determine theactual feature value using the feature-appropriate units (e.g., time ofa beat in fractional seconds, pitch of a melody at a specific momentusing hertz, amplitude in decibels, and the like).

According to some embodiments, FIGS. 6-10 provide disclosure ofnon-limiting example embodiments of Step 504's analysis for processingan audio file. In some embodiments, FIGS. 6-10 provide embodiments forprocessing an audio file to determine one or more overlays or othereffects that are suitable for overlaying with other audio content (e.g.,voiceover, audio identifiers, sound effects, sonic branding, and thelike, for example). Some embodiments use artificial intelligence (AI)and/or digital signal processing to identify regions of a song or otherfile that are appropriate for overlay of other content (voiceover, soundeffects, music, and the like) and extracting features (time markers, andthe like) that help guide the overlay (e.g., CNNs as discussed above).Some embodiments include substantially or exactly “hitting the post”using overlaid content, as discussed below, where “posts” correspond tosignificant moments in a particular song (or type of audio), forexample, which should be allowed to be rendered in an interrupted orunmodified manner (e.g., the guitar solo in “Free Bird” by LynyrdSkynyrd). In some embodiments, portions of audio files that areidentified as “posts” are tagged, whereby modification of the audio fileand/or modification of its playback is restricted.

By way of background, digital and terrestrial radio stations play musicto listeners. Audio is typically inserted in between or dubbed overportions of the music. For example, disc jockeys (DJs) may speak overthe beginning portion of the song or may play a prerecording of audiocontent that overlaps with a portion of a song. In some domains, DJs,fitness instructors, or other persons may overlay their voices oversignificant portions of songs. In terrestrial radio, determining whereit is safe to perform an overlay at the front of a song is done by ahuman who “tags” a particular song with one or more time markers thatindicate “posts” or other “sonically significant”—for example, momentswhere a DJ should stop talking. Some songs may be unsuitable forvoiceover or other overlays in their entirety (e.g., radio edits thatremove portions of original files from where availability to overlaycontent is removed or filtered out). These human-performed analyses canbe complex, time-consuming and subjective, leading to inaccurate or lessdesirable experiences and outcomes.

As discussed herein in relation to at least FIGS. 6-10 , the disclosedframework addresses these technical shortcomings by providing systemsand methods for identifying regions of audio files (or songs, usedinterchangeably) that are considered “safe” for overlay of differenttypes of content (e.g., with the human voice being the primary one insome embodiments) and then, within those regions, the moments to whichthe audio is aligned will sound better—more musical or artful.

According to some embodiments, such overlay regions can be identified inan audio file. In some embodiments, the overlay regions can beidentified in a video as well, where audio is being provided as abackground, for example. However, for purposes of this disclosure, anaudio file will be discussed; however, it should not be construed aslimiting, as one of skill in the art would recognize how the disclosedsubject matter can be applied to different media file types.

For example, an audio file can represent a song. Some embodimentsidentify a musical moment where the song's melody, beat, vocals orinstrumentals, and the like, become prominent or where they becomesignificantly less prominent. In some embodiments, the segmentation anddownbeat analysis can be leveraged to identify an exact moment(s) forthe most “natural” entrance and exits of overlay material, as discussedsupra. Between these musical moments, in some embodiments, the audioprocessor identifies candidate sections that are deemed appropriate foroverlaying additional audio content. The audio content may bepre-recorded, taken from a live source or “rendered” (either prior or inreal-time) using text to speech or other suitable techniques.

FIG. 6 depicts a networked environment 600 according to someembodiments. The networked environment 600 includes a computing system601 that is made up of a combination of hardware and software, asdiscussed above in relation to FIGS. 1-3 .

The computing system 601 includes a database 603, a streaming service611 and an audio (or music, used interchangeably) processor 613. In someembodiments, as discussed in detail below, system 601 can further oralternatively include, mixdown agent 614 and content generator 615 (thefunctionality of which are discussed in more detail below in relation toFIGS. 13-18 ). The computing system 601 may be connected to a network616 such as the Internet, intranets, extranets, wide area networks(WANs), local area networks (LANs), wired networks, wireless networks,or other suitable networks, and the like, or some combination thereof(as discussed above in relation to at least FIG. 1 ).

The computing system 601 may comprise, for example, a server computer orany other system providing computing capability. Alternatively, thecomputing system 601 may employ a plurality of computing devices thatmay be arranged, for example, in one or more server banks or computerbanks or other arrangements. Such computing devices may be located in asingle installation or may be distributed among many differentgeographical locations. For example, the computing system 601 mayinclude a plurality of computing devices that together may comprise ahosted computing resource, a grid computing resource and/or any otherdistributed computing arrangement. In some cases, the computing system601 may correspond to an elastic computing resource where the allottedcapacity of processing, network, storage, or other computing-relatedresources may vary over time. The computing system 601 may implement oneor more virtual machines that use the resources of the computing system601.

Various applications and/or other functionality may be executed in thecomputing system 601 according to various embodiments. Also, variousdata is stored in the database 603 or other memory that is accessible tothe computing system 601. The database 603 may represent one or moredatabases 603.

The streaming service 611 and audio processor 613 mentioned above arecomponents executed on the computing system 601. These components maygenerate data and store the data on the database 603 and/or access thecontents of the database 603. The streaming service 611 may be anapplication implemented on one or more webservers that enable users tosubscribe, create, edit, and manage stream audio (e.g., digital radiostations). The streaming service 611 receives user input and generatesan encoded audio stream that is transmitted over the network 616 forplayback.

The audio processor 613 may comprise a software application or modulethat may communicate with the streaming service 611. The audio processor613 may employ one or more APIs or other interfaces to plug into thestreaming service 611, receive control commands and data from thestreaming service 611 and generate output data that is transmitted tothe streaming service 611.

In some embodiments, the data stored in the database 603 includes anaudio library 622, user profiles 625, and overlay content. The audiolibrary 622 may comprise audio files. As discussed below, the audiolibrary may include portions or segments for storing components, slicesor other portions of an audio file (e.g., a primary library and asecondary library).

According to embodiments of the instant disclosure, an audio file may bea song file, audio recording, or any other audio file. In someembodiments, the audio file may include any or all types of metadatasuch as artist, title, album information, chapter information,descriptive and relational tagging, and the like.

The user profiles 625 include data for various user accounts managed bythe streaming service 611. User profiles can include similar informationas discussed above in relation to database 420 in FIG. 4 . A useraccount may include a user name, password, credentials, and othersubscription information. The user profiles 625 may include the audiopreferences of a user, such as, an identification of preferred songs,radio stations, and other experiential preferences. In some embodiments,the overlay content 628 may be a library of audio files containingbranding and advertising content.

In some embodiments, the overlay content 628 may be a library of audiofiles containing retail branding and advertising content for in-store orother suitable environments. In some embodiments, each item making upthe overlay content can correspond to metadata, such as, but not limitedto, the duration of the item, size of the item, and the like.

The networked environment 600 also includes one or more client device(s)633. A client device 633 allows a user to interact with the componentsof the computing system 601 over the network 616. A client device 633(as discussed above in relation to at least FIGS. 1-3 ), may be, forexample, a networked speaker, a cell phone, laptop, personal computer,mobile device, or any other computing device used by a user. The clientdevice 633 may include an audio player 637 comprising an applicationsuch as a web browser or mobile application that communicates with thestreaming service 616 to select and receive audio streams. The clientdevice 637 may also comprise a radio receiver such as, for example, anAM/FM receiver for receiving terrestrial broadcasts.

Next, a general description of the operation of the various componentsof the networked environment 600 is provided in accordance with someembodiments. Through a client device 633, a user may subscribe to astreaming service 611 and specify a preference to an audio stream. Theaudio service 611 selects various audio files from the audio library 622and assembles them in serial order into an audio stream that is thentransmitted over the network 616 to a client device 633. The streamingservice 611 may dynamically create a playlist of audio files to bestreamed in a particular order. The playlist may include the currentlystreamed audio file, the subsequently streamed audio file, andpotentially additional audio files to be streamed in order. As thestreaming service 611 moves down the playlist, it prepares the audiofiles to be transmitted into a digital stream.

In some embodiments, the audio player 637 of a client device 633receives the audio stream, decodes it, and plays it back through one ormore speakers in communication with the client device 633. The user mayprovide input to the streaming service 611, which can include suchactions as, but not limited to, skipping to the next track, pausing,changing stations, providing feedback regarding an interest (e.g.,“like” or “dislike”), and in some embodiments, as discussed below, canprovide parameters to alter the output (e.g., change volume, energylevel, speed of playback, aggressiveness of overlaying, factors relatingto the personality or overall perception of the output, and the like).In response, according to the disclosed functionality, the streamingservice 611 may access the audio library 622 to create an updated audiostream in response to the user input. In some embodiments, some or allof the parameters are adjusted solely by the streaming service or itsvendors to provide a desired user listening experience.

Some embodiments of the present disclosure are directed to an audioprocessor 613 that fundamentally enhances the functionality provided bya conventional streaming service. The audio processor 613 processes theaudio files to generate an output file. The output file may then betransmitted to the streaming service 611, whether it is then transmittedto the client device 613 or it may be transmitted to the client device613 directly. The following provides detailed examples of thefunctionality of the audio processor 613. Embodiments of the audioprocessor 613 are described in greater detail below with respect to theremaining figures.

FIGS. 7A-7B are diagrams illustrating examples of an audio file beingprocessed by the audio analyzer in the networked environment of FIG. 6 ,according to some embodiments of the present disclosure. FIG. 7A showsan embodiment of an audio file 700 that is processed by the audioprocessor. The audio file 700 includes a song that is composed withvarious musical components. The audio processor 613 receives the audiofile 700 and identifies musical moments (including, but not limited to,posts) and overlay-eligible regions in the audio file. A musical momentmay be, for example, when a quiet introduction of a song ends and theprimary melody begins, where the vocals of a song begin or end, where aprimary instrument begins or ends, when a particular verse or chorusbegins or ends, when the song winds down and an outro begins.

As discussed herein, the processor 613 can identify these portions(e.g., melody and/or pitch moments/portions) via a CNN or at least twoCNNs, where each CNN is trained to focus on a specific portion, orsatisfy a predetermined confidence rate of retrieval/detection. Thus,portions of a song where singing, for example, may be occurring can beavoided as being overlaid (or trimmed as part of a radio edit, asdiscussed below). For example, when processor 613 determines that aportion of an audio file (using an extracted vocal stem) has anamplitude above a threshold level, then this can be an indication thatsinging is occurring, and this portion (e.g., buffered by apredetermined number of bits, in some embodiments), can be tagged as apost to avoid during a mixdown.

Some embodiments can be used for a single file alone and for multiplefiles that are being played in sequence (and transitioned between them).Some embodiments include additional logic where, for example, file A andB are overlain and then a determination is made to add voiceover contentC over the period of the overlap. In some embodiments, the attributes ofA and B are evaluated during the overlap period to have a deeperunderstanding of the eligibility and timing of the C element and thebehavior of the externally or internally generated transition may bealtered by this new end state. In some embodiments, after identifyingthe musical moments and/or overlay-eligible portions, the audioprocessor segments the audio file into one or more candidate sections705 and one or more restricted sections 708. In some embodiments, thesections 705, 708 are defined by the transitions serving as the boundarybetween segments. A candidate or overlay-eligible section can be markedwith confidence scores or levels 705 and can be a portion of the songwhere it is deemed acceptable to overlay the song with overlay content628. A restricted or overlay-ineligible section 708 is a portion of thesong where it is deemed unacceptable to overlay the song with overlaycontent 628. These acceptability and unacceptability determinations canbe performed in a wide variety of manners, but algorithmicdeterminations are used in some embodiments.

According to some embodiments, the musical moments and/or portions ofthe file may be classified as overlay eligible or overlay ineligible. Insome embodiments, an overlay ineligible portion corresponds to a “hardpost”—with no talking or other voiceovers or overlays being allowed pastthis point at the front of the song. In some embodiments, “soft posts”are musical timestamps within the overlay eligible or “safe” regionsthat are “alignment opportunities” with the effect of the voiceover (orother audio element) being more artfully integrated in with the music.In some embodiments, the soft posts represent things like musicalinstrument entries and/or exits or structural changes in the compositionitself.

The audio processor 613 may be configured in various ways to identifycandidate sections 705 and restricted sections. In some embodiments,audio processor 613 may receive audio files that are manually tagged toindicate the position of the transitions or character of the content(e.g., “has voice” versus “instrumental”). The audio processor 613 maycomprise a classifier that is trained according to the manually taggedaudio files to classify additional audio files. In this respect, theclassifier is trained using training data to generate overlays for newaudio files based on tagged samples. For example, the audio processor613 may divide a waveform into segments and then classify those segmentsusing a binary or ordinal classifier.

The audio processor 613 may implement artificial intelligence algorithmsto analyze the waveform of the audio file to identify overlay eligible(“safe”) and overlay ineligible (“unsafe”) portions (with or without anassociated confidence score for each region). For example, the audioprocessor 613 may locate the time positions in the waveform where theamplitude suddenly increases and is sustained for a predetermined amountof time. The audio processor 613 may also or instead locate the timepositions where the waveform transitions from periodic to moreirregular.

According to some embodiments, the identification of eligible and/orineligible portions can involve, but is not limited to, determining aconfidence value for these portions (as mentioned above); and when theconfidence value is at or above a threshold value, then they can bemarked accordingly.

In some embodiments, the manually tagged audio files may be used tosupplement or override the overlay regions identified by employing thealgorithms discussed above (using explicitly tagged audio regions or alist of one or more timestamps that are used for synchronizing thedisplay of lyrics). For example, AI algorithms and machine learningalgorithms (e.g., CNNs) might be insensitive to culturally significantsections of a song. Such culturally significant sections can include,for example, but are not limited to, song sections considered classics,or other desired criteria or attributes can be used that contribute toor detract from overlay suitability. For example, such sections can be asection that, but its features, is safe for overlay but would beculturally insensitive to do so (e.g., a quiet vocal at the end of asong, a special moment in a guitar solo during a fade-out, and thelike). Therefore, pre-tagged songs may override the decision makingprocess as a mechanism to create certain desirable exceptions forspecific songs. In some embodiments, many factors can contribute totagging confidence scores or levels, as discussed herein.

In some embodiments, candidate sections 705 may be identified asportions that can be “trimmed”—either removed (by modifying the audiofile) or tagged as a portion to automatically skip when rendering fromeither the beginning or end of an audio file. That is, some songs haveeither beginning or end portions that can be removed without impactingthe listening experience of the song. These are known as “radio edit”portions that DJs would either skip over (e.g., start a track at a pointafter the beginning of a song) or speak over (as it played in thebackground). Thus, in some embodiments, according to the mechanismsdiscussed herein, the audio file can be analyzed (e.g., via CNNsdetection of a certain type of content and/or lack thereof (e.g., aportion that is inconsequential to the audio content), and theseportions can be identified as an alternative embodiment of candidatesections 705, whereby engine 400 can remove or tag the portion so that a“radio edit” is achieved. This modified audio file can then be stored ina database, which can be performed according to the embodiments ofstoring audio information, as discussed below.

The audio processor 613 may also employ the capabilities of a digitalsignal processor (DSP) to identify candidate sections 705 and restrictedsections 708. The DSP may be configured to determine overlay eligibleand ineligible portions using melodic identification, identification offrequencies in the human vocal range, identification of significantinstrumental sections, identification of human speech or singing, or awide variety of other identification techniques (e.g., identificationsegmentation and downbeats, as discussed above) further detailed inpatent applications incorporated by reference herein. Some embodimentsuse source-separation and then analysis of those files which may includeextracting (e.g., using digital signal processing (DSP) and machinelearning (ML), for example) the vocals from the track and then analyzingthe timing and other characteristics of that isolated track.

Some embodiments can analyze files comprising “stems” (a multi-channelmix of the song where the system can enable only particular musicalinstruments or performers). The aforementioned DSP/ML can also be usedto extract the stems in some embodiments.

FIG. 7B shows the selection of overlay content 628 based on theidentified candidate sections 205. For one or more candidate sections705, the audio processor 613 selects overlay content 628, such as anaudio clip. According to some embodiments, the audio processor 613selects the overlay content 628 by matching the duration of the overlaycontent to the duration of the candidate section 705. In someembodiments, characteristics of the candidate section that help toidentify the appropriate overlay element can include, withoutlimitation, duration, tempo, key, energy level, energy, instrumentation,structural position, and the like. In other embodiments, the overlaycontent may be a live audio stream or any other audio signal. In someembodiments, the overlay content can comprise branding content which canbe well-integrated into the resulting content and output file. In someembodiments, content can be produced in a wide range of lengths andother variations, and the processor 613 can select the best fittingcontent for the particular application. Some embodiments can identifyregions that are “safe with modification” which could include techniqueslike reducing the amplitude of the source material, removing vocals fromthe material, or even remixing the material in other ways.

FIG. 8 is a diagram illustrating an example of an output file in thenetworked environment of FIG. 6 , according to various embodiments ofthe present disclosure. FIG. 8 shows a processed audio file 800 that isgenerated by inserting a mixed section 803 into the processed audio file300. The mixed section is generated by mixing the overlay content 628with the candidate section 705. There may be one or more instances ofmixed sections 303 inserted into the processed audio file 800. In someembodiments, the insertions are naturalistic, artful and respectful ofcontext and other factors. In some embodiments, the disclosed systemsand methods utilize analysis of the audio sections that are to beoverlain and analysis of the overlaying audio to match the two sections(e.g., by selecting a “low energy” voice read for a “low energy” sectionof music).

According to some embodiments, in fitness music applications, a clips ofa fitness instructor's voice commands or instructions may be insertedinto the process audio file 800 so that the audio file plays andincludes interruption in only portions of a song that are deemedacceptable. Some embodiments provide a seamless and pleasing audioexperience balancing the need to provide fitness instructions whilepreventing overlaying instructions over portions of the audio file whichshould not be overlain.

FIG. 9 is a flowchart illustrating an example of the operations of theaudio processor 613 executed in the networked environment 600 of FIG. 6, according to various embodiments of the present disclosure. It isunderstood that the flowchart of FIG. 9 provides merely an example ofthe many different types of analyses and determinations of functionalarrangements that may be employed to implement the operation of theaudio processor 613 described herein (e.g., an embodiment of Step 504,as discussed herein).

In Step 902, the audio processor 613 obtains an audio file 300. Theaudio file 300 may be obtained from an audio library 622 or may beextracted from an audio stream or any other suitable source. Forexample, an audio file may be uploaded by service 611 to an FTP site,where process 613 retrieves it for processing.

In Step 904, the audio processor 613 performs processing to detectoverlay eligible and overlay ineligible regions. As discussed above, theprocessing can involve analyzing the audio file using, for example, aCNN.

In Step 906, the audio processor 613 identifies, based on the processingof Step 904, time stamps of segments for overlay content 628. Forexample, the audio processor 613 identifies candidate sections 705 andrestricted sections 708 in between the identified musical moments orregions. The audio processor 613 may generate a list of the timestampsfor each section 705, 708.

In Step 908, the audio processor 613 selects overlay content 628 basedon the timestamps in accordance with some embodiments. In someembodiments, the selection can be based on application of a CNN. Theoverlay content 628 might be selected as a targeted advertisement basedon the listener or any other audio clip taken from a library, and theduration of the overlay content 628 is selected to match the duration ofthe candidate section 705. In some embodiments, the overlay content maybe processed or created to be longer or shorter to match the duration ofthe candidate section 705. This may include, for example, timestretching/shrinking or clipping operations.

In Step 910, the audio processor generates a processed audio file 800having the overlay content 628 in accordance with some embodiments. Insome embodiments, the overlay content is mixed with a selected candidatesection 705 to create a mixed section 803. The mixed section 803replaces the candidate section 705 in some embodiments. Thus, theprocessed audio file 800 is the same as the original audio file 700except that it includes overlay content 628 that is mixed into the audiofile at a time range that improves the listening experience in someembodiments.

FIG. 10 is a schematic block diagram that provides one exampleillustration of a computing system 601 of FIG. 6 according to variousembodiments of the present disclosure. The computing system 601 includesone or more computing devices 1000. Each computing device 1000 includesat least one processor circuit, for example, having a processor 1003 andmemory 1006, both of which are coupled to a local interface 1009 or bus.To this end, each computing device 1000 may comprise, for example, atleast one server computer or like device. The local interface 1009 maycomprise, for example, a data bus with an accompanying address/controlbus or other bus structure as is known in the art.

Stored in the memory 1006 are both data and several components that areexecutable by the processor 1003. In particular, stored in the memory1006 and executable by the processor 1003 is the streaming service 611and audio processor 613. Also stored in the memory 1006 may be adatabase 603 and other data such as, for example, audio library 622,user profile 625, and overlay content 628. In addition, an operatingsystem may be stored in the memory 1006 and executable by the processor1003.

In some embodiments, as discussed in detail below, system 601 canfurther or alternatively include, mixdown agent 614 and contentgenerator 615.

Although the streaming service 611 and audio processor 613 (and mixdownagent 614 and content generator 615) described herein may be embodied insoftware or code executed as discussed above, as an alternative the samemay also be embodied in dedicated hardware or a combination ofsoftware/hardware and dedicated hardware. If embodied in dedicatedhardware, each can be implemented as a circuit or state machine thatemploys any one of or a combination of a number of technologies. Thesetechnologies may include, but are not limited to, discrete logiccircuits having logic gates for implementing various logic functionsupon an application of one or more data signals, application specificintegrated circuits (ASICs) having appropriate logic gates,field-programmable gate arrays (FPGAs), or other components, and thelike. Such technologies are generally well known by those skilled in theart and, consequently, are not described in detail herein.

In some embodiments, the audio processor 613 (and mixdown agent 614 andcontent generator 615) may also comprise software or code that can beembodied in any non-transitory computer-readable medium for use by or inconnection with an instruction execution system such as, for example, aprocessor 1003 in a computer system or other system. In this sense, thelogic may comprise, for example, statements including instructions anddeclarations that can be fetched from the computer-readable medium andexecuted by the instruction execution system.

Further, any logic or application described herein, including thestreaming service 611 and audio processor 613 (and mixdown agent 614 andcontent generator 615, as discussed below) may be implemented andstructured in a variety of ways. For example, one or more applicationsdescribed may be implemented as modules or components of a singleapplication. Further, one or more applications described herein may beexecuted in shared or separate computing devices or a combinationthereof. For example, the software application described herein mayexecute in the same computing device 1000, or in multiple computingdevices in the same computing system 601. Additionally, it is understoodthat terms such as “application,” “service,” “system,” “engine,”“module,” and so on may be interchangeable and are not intended to belimiting.

Turning back to FIG. 5 , according to some embodiments, the analysis ofStep 504 can involve analysis of the audio file, and subsequent storagein a dedicated database(s) for retrieval at a later time. The storage,which is based on the content and/or attributes of the audio within theaudio file, is the basis from which the organization and cataloging isperformed. Thus, having performed the analysis of Step 504, Process 500proceeds to Step 506 where the audio file is organized and catalogedaccordingly.

By way of a non-limiting example, according to some embodiments, turningto FIG. 11 , Process 1100 provides an embodiment for the determination,detection, retrieval, extraction or otherwise identification ofattributes, characteristics, features, deep descriptors, and the like,or some combination thereof, of the audio file. Step 504, and itssub-steps: Steps 1102-1108 of Process 1100, discussed herein. Then,based on this information, the audio file, and its determinedinformation, are cataloged accordingly, as in Step 506 (and itssub-steps: Steps 1110-1114 of Process 1100). Thus, in some embodimentsas discussed below, Process 1100 provides embodiments for theperformance of Steps 504-506 of Process 500 of FIG. 5 .

Process 1100 begins with Step 1102 where the identified audio file(s)from Step 502 is parsed, from which portions (e.g., slices) of the audiofile are identified. Such portions, for example, can include, but arenot limited to, samples of the audio, normalized versions of the audio,segmentation of the audio, extracted audio and melodic portions, and thelike.

In Step 1102, the parsed files are analyzed. As mentioned above, suchanalysis can involve analysis and identification of data/metadata by anyknown or to be known audio (or media) file analysis technique,algorithm, classifier or mechanism, including, but not limited to, ANNs,CNNs, computer vision, Bayesian network analysis, Hidden Markov Models,data mining, feature vector analysis, logical model and/or treeanalysis, data mining, and the like.

Based on the analysis, information related to, but not limited to,melodic features, tempo regions, amplitudes, beats per minute (BPM),fade ins/outs, features of individual stems (using source separation),dominant frequency ranges, structure, beat positions, onsets, harmonics,speakers/singer quantity, background noise, energy level, pitch, silencerates, duration, sonic genre classification (multiple classificationswith or without weights), loudness, key, meter, gender of vocals (maleor female), arrangement (music with vocal or instrumental), mood(happiness and sadness), character (acousticness and electronicness),danceability, harmony (tonal or atonal), attitude (aggressiveness andchillness), environmentalness (music or environmental sounds),environmental sonic genre (multiple classifications with or withoutweights), and/or any other acoustic or DSP metric, value orcharacteristic that is identifiable from an audio file, or somecombination thereof, can be determined, derived, extracted or otherwiseidentified, as in Step 1106.

In some embodiments, for example, voice portions, portions attributed tocertain instruments (e.g., drums), and/or other information related totypes of audio characteristics (e.g., melody, volume, rhythm, and thelike), can be extracted from the portions as a by-product or result ofthe computerized analysis.

In some embodiments, the audio information can further include theinformation provided upon identification of the audio (e.g., from Step502—for example, the type and/or identifier). This, as discussed below,can be is used to route information to specific databases and/or providean indication of a format type of storage in such databases.

According to some embodiments, the audio features, characteristicsand/or attributes of the audio file identified from at least FIGS. 6-10(and FIGS. 13-17 discussed below), can also be identified in Steps1104-1106, and form the basis of such audio information, which can befurther processed, as discussed below. For example, informationindicating “posts”, overlays, mixdown portions, and the like, asdiscussed above, can be identified and/or extracted from the audio file.

In Step 1108, the type of audio information is analyzed. In someembodiments, this analysis dictates or forms the basis for whichdatabase (e.g., identity and type of database), or which portion of amultidimensional database, the audio information is stored in, and inwhich manner, form and quantity of the audio information is storedtherein. For purposes of this discussion, multiple databases arediscussed; however, one of skill in the art would recognize that amultidimensional database would function in a similar manner.

In Step 1110, the appropriate database for the audio information isidentified. For example, for the vocals and/or other “content” of anaudio file, a content database (e.g., a content digest) is identified.In some embodiments, this type of database is capable of being subjectto a query that searches for content based on a variety of factors thatcan include, but are not limited to, a search string, context variables,using a key-value pair as the basis for identifying and retrieving theaudio file's vocal information, for example, and the like.

In Step 1112, the format of the storage within the identified databaseis identified. For example, if the database is a vector database forstoring the audio features as a n-dimensional feature vector, then thisinformation would serve as the format identified in Step 1110. Anexample of this is provided below in relation to FIG. 12 .

In another non-limiting example, if the content database only storeskey-value pairs as references/pointers to remotely located content, thenthis information can be identified and leveraged, as discussed below inrelation to FIG. 12 . In some embodiments, the storage of key-valuepairs can correspond to individual or a set of features. In someembodiments, the storage of key-value pairs can be associated with astored, compressed version of a content descriptor record as a value,with a contextualized client identifier as the composite key (e.g.,version+partner+type+client identifier).

In Step 1114, the audio information, either an appropriate portion or aversion of it, is formatted and stored accordingly. In some embodiments,Step 1114 can include identification of particular portions of audioinformation for storage in particular databases. That is, for example,as mentioned above, the content (e.g., vocals) of the audio informationcan be extracted and formatted as a representative key-value pair, whichcan be stored in a content database that is capable of being queried. Inanother example, the deep features are identified as being suitable fora vector database. Therefore, these deep features of the audio can besubject to known or to be known vectorization techniques, and stored asa feature vector in a vector database.

In some embodiments, the storage of Sep 1114 can involve enabling accessto a remote location for a user (e.g., a third party provider) to accessto analyzed audio file portions/data (e.g., an FTP site or any othersuitable repository).

FIG. 12 provides Process 1200 for further processing of an audio file.Such processing can be a sub-process of Steps 504 and 506's operations,as mentioned above. In some embodiments, Process 1200 can be executed aspart of Processes 500 and/or 1100, or can be a separate process thatexecutes as a mechanism for analyzing and cataloging audio files andtheir data/metadata.

Process 1200 begins with Step 1202 where the audio file is parsed (andanalyzed) in a similar manner as discussed above in relation to Steps504 and 1102. In Step 1204, a set of predetermined portions of the audiofile are identified, and such portions correspond to a predeterminedtime period of the audio file.

In some embodiments, Step 1204 involves performing fingerprintingalgorithms (e.g., hash functions) that enable the reduction of largedata files to shorter, representative files (e.g., MBs to KBs of data)which survive and are capable of being subject to encoding at differentbit rates.

For example, two portions of the audio file can be identified, and theycorrespond to the first n seconds (e.g., 2 minutes or 120 seconds) ofthe audio file, and the last n seconds of the file. These are referredto as “fingerprints” and “toeprints”, respectively. An example of suchsections can be viewed as candidate sections 705 at the beginning(front) and end of audio file 700 of FIG. 7A.

In Step 1206, these portions are analyzed, and as a result, in Step1208, metadata related to the portions features and/or attributes areidentified (e.g., the metadata of each portion). In some embodiments,Step 1208 also further identifies the distance between the end of thefirst portion (e.g., fingerprint) and the beginning of the secondportion (e.g., toeprint).

In some embodiments, the analysis of Step 1206 involves featureextraction. In some embodiments, the analysis can be performed in asimilar manner, and according to similar mechanisms discussed above inrelation to Steps 504 and 1104, discussed above.

Continuing with Process 1200, in Step 1210, a fingerprint database ofhashes (or fingerprints) is identified (e.g., using MusicBrainz™ orother similar resources in some embodiments, as a non-limiting example).In some embodiments, this identification leads to a search of thedatabase based on the information from the fingerprints (or hashes) ofthe audio file (e.g., from Step 1204).

As a result of the search, groupings (or clusters, used interchangeably)are determined. The groupings can be based on a time-synched matchingbetween the hashes of the audio file and the data stored in thefingerprint database identified in Step 1210. The groupings, in someembodiments, involve, but are not limited to, IDs for i) fingerprints;ii) groups; and/or iii) families.

In some embodiments, the fingerprint ID information, which provides aunique ID, corresponds to information identified from or associated withthe audio file. In some embodiments, the group ID provides an identifierthat indicates non-duplicative (e.g., subject to a de-dupe) data forother files having the same audio. For example, the identification froma collection of audio of the same music track (e.g., the recorded trackversus a live version, track recorded by artist X and the same trackrecorded by artist Y, for example). In some embodiments, the group IDcan reference songs that comprise the same content, but are offset bycertain time stamps (e.g., the same song on differentcompilations/albums, where the “time between tracks” may be different inorder to realize consistent musical flow that is specific to therespective compilation or album). In some embodiments, the family IDcorresponds to similar recordings, and/or those that are from the samealbum, artist, time period, and the like, and/or correspond to the samesong (e.g., song recorded by different artists).

In some embodiments, similarity data, from group IDs and/or family IDs,can be utilized as a way to reduce the computational load on performingthe disclosed analysis. For files within the same grouping (e.g., groupID and/or family ID), one file may be analyzed, and its findings can beapplied to its corresponding counterpart songs within a grouping. Thisenables a grouping of files to effectively be analyzed via theprocessing of a single file for the group. In some embodiments, furtherprocessing may be required to gather/collect basic information about theother files in the group. In some embodiments, when combined with anoffset for the group (e.g., a fingerprint offset), audio features forthe group as well as each individual file can determined therefrom.

This information is then stored in a fingerprint database associatedwith engine 400. Step 1214. This information can be stored within alook-up table (LUT) and/or as vector information, as discussed above. Insome LUT embodiments, engine 400 can utilize an inverted index toidentify portions of the fingerprint, such that quality matches for agrouping is derived from the number of fingerprint segments matched by apiece of content. Thus, the more it matches (ultimately in sequence),the more similar the files are considered to be. In some embodiments,the hash information can be stored as key-values, as discussed above.

For example, Steps 1212-1214 involve comparing hash information from theaudio file (determined from Step 1208) to information stored in thedatabase identified in Step 1210. This comparison, which can beperformed via neural network analysis (e.g., CNN), for example, canindicate similarities of the audio data/metadata, which can lead to thegroupings discussed above (items i, ii, and/or iii of the groupings).

In some embodiments, engine 400 can identify duration of the audioelements, distance between the fingerprint and toeprint, and presence ofsignal on the outside bounds of the fingerprint offsets. In someembodiments, this can be utilized to identify if there is audio materialbefore the fingerprint or after the toeprint which would cause the fileto be unique compared to others with similar fingerprints and toeprints.

Turning back to FIG. 5 , having organized and cataloged the data andmetadata for the audio file(s), Process 500 turns to the processing ofconfiguring and managing the audio file (and other files), as in Step508.

According to some embodiments, the processing performed in accordancewith Step 508 is detailed according to the following disclosure of FIGS.13-15 , which provides systems and methods for processing files forplayback.

By way of background, streaming services (e.g., service 611) allow aclient device to select a digital station and receive an encoded audiostream that the client device can decode and play back via one or morespeakers or other devices. Such streaming services can include aserver-based application that selects different audio files from alibrary and transmits them for playback in serial order.

As discussed herein in relation to at least FIGS. 13-15 , the disclosedframework addresses these technical shortcomings by providing systemsand methods for a cloud-based or client-based mixdown agent 614 that canprocess audio files before they are streamed as part of an audiostreaming service or otherwise distributed as desired. The mixdown agent614 is illustrated in FIGS. 6 and 10 , as discussed above, whereby itsfunctionality is discussed herein.

In some embodiments, the mixdown agent 614 can process the tail portionof an initial audio file with the head portion of a subsequent audiofile to generate mixed versions of the same so that the initial audiofile and subsequent audio file are played back seamlessly (withoutperceptible interruption or completely uninterrupted as desired.) Theseembodiments can provide gapless stitching of audio files. Someembodiments can use heads and tails and then leverage gapless,sequential playback to build a contiguous audio experience (with head,middle, and tail sequences). Some embodiments can also use full tracksand render a succession of full files that, when played gaplessly, givethe impression of a contiguous live stream. Some embodiments can alsooutput a continuous live stream of audio that is encoded and deliveredas a live experience (using various streaming techniques, such as, butnot limited to, Real-Time Streaming Protocol (RTSP) or HypertextTransfer Protocol (HTTP) Live Stream (HLS), for example). Someembodiments can also be used to simply produce completed audioexperiences—for example, a fully produced ad or something longer like apodcast. Some embodiments combine multiple elements and produce theminto a single file, feed or stream. Some embodiments reduce the numberof files required on the user end. By way of a non-limiting example,according to some embodiments, two or more audio files, and some or alltheir relevant information, can be combined from multiple files andsaved to fewer files, or even one file.

According to some embodiments, as discussed herein, the mixdown agent614 performs one or more of the following functions: overlayingadditional content during the mixdown process, inserting metadata intothe output file, and inserting one or more index points (also known asskip stops) which have metadata associated with them in the output file.Some embodiments include the ability to process smaller portions ofaudio files, thereby reducing computing resources demands. In someembodiments, selecting specific portions of the audio files to processby the mixdown agent 614 provides improved listening experiences asopposed to arbitrarily crossfading consecutive audio files.Additionally, some embodiments provide more than just mixdownfunctionality by providing a fully featured production agent that doesmixing, timing, overlays, processing, and the like. Some embodiments canrender one or more personalized advertisements from multiple audioelements.

According to some embodiments, the mixdown agent 614 can comprise asoftware application or module that communicates with the streamingservice 611. In some embodiments, the mixdown agent 614 can employ oneor more APIs (or other suitable interfaces) to plug into the streamingservice, receive control commands and data from the streaming service611 and generate output data that is transmitted to the streamingservice 611 (as illustrated in FIG. 6 , for example).

According to some embodiments, the functionality described with respectto the mixdown agent 614 can be implemented in a client device. In someembodiments, the functionality can be implemented via a server,collection of servers, and/or a distributed CDN.

In some embodiments, a mixdown agent 614 can supplement and/or replacesome of the functionality provided by the streaming service 611 (asdiscussed above). In some embodiments, mixdown agent 614 can receive theplaylist constructed by the streaming service 611. In some embodiments,the mixdown agent 614 can receive the audio files and any overlaycontent 628 as an input. The mixdown agent 614 processes the audio filesto generate an output file. The output file can then be transmitted tothe streaming service 611, whether it is then transmitted to a clientdevice. FIG. 13 , discussed below, provides detailed examples of thefunctionality of some embodiments of the mixdown agent 614.

FIG. 13 is a flowchart illustrating an example of the operations of themixdown agent 614 according to some embodiments of the presentdisclosure. It is understood that the flowchart of FIG. 13 providesmerely an example of the many different types of functional arrangementsthat can be employed to implement the operation of the mixdown agentdescribed herein. In addition, the flowchart can be implemented asmodules that can be configured to facilitate orchestration. Someembodiments can run on clients as well—iOS, Android, M, PC, raspberrypi, and the like. Some embodiments can provide some or all of thefunctionality described herein in a retail or other environment where asmall (for example, US$30) computer is mixing down and producing theaudio that you hear in the store or other location. Some embodiments canalso be implemented in a non-network environment or in an environmentthat has periodic access to the network. Some embodiments can provide awide variety of production functions including, without limitation, adgeneration, podcast generation, and the like.

Process 1300 begins with Step 1302, where the mixdown agent 614 obtainsmixdown parameters. In some embodiments, the mixdown parameters are a“recipe” for how the mixdown agent should process the inputs. Forexample, in some embodiments, the mixdown parameters can include anidentification of one or more audio processes. An audio process can be,for example, a frequency filter, limiter, a cross fade process, anattenuation process, an equalization process, a dynamics processing, orany other audio process. The parameters can include instructions such aswhether to insert one or more skip stops or metadata into an output.

In Step 1304, the mixdown agent 614 obtains audio data. The audio datacan be multiple audio files from an audio library 122 or can be an audiostream compiled from multiple audio files. According to someembodiments, the audio data includes a first audio item (or audio file,used interchangeably) (e.g., a song) and a second audio item (e.g.,another song) to be played in consecutive order. In addition, the audiodata can include overlay content 628.

In Step 1306, the mixdown agent 614 generates mixdown output files. Insome embodiments, the mixdown output files are generated by processingthe tail portion of a first audio item and the head portion of a secondaudio item. For example, Step 1306 can include a “slicer” operation, asdiscussed above, that parses and extracts separate portions (or clips)from input audio items. In some embodiments, each clip is a portion(e.g., head, middle, tail) that can be handled as a separate file.

Some embodiments join multiple items into a larger whole or simply tobuild a single item that consists of multiple parts. By way of anon-limiting example, an audio ad campaign can be produced by someembodiments where an announcer reads a car manufacturer ad, then readsall the names and addresses for every local dealer, and a producer putstogether a collection of different genres of background music. Someembodiments can create specific advertisements for every dealer andevery possible music format by following the instructions passed to itto combine the relevant elements (e.g., main ad read, West Texas dealeraddress, and country background music.)

In Step 1308, the mixdown agent 614 generates an output stream. In someembodiments, mixdown agent 614 combines the audio files into an outputstream that can be received and played by a client. The mixdown agent614 can encode the output stream. For example, the mixdown agent mayapply data compression to prepare it for transmission over a network116. Such transmission can involve sending to a requesting user/entity,or hosting on a network resource location for retrieval.

In Step 1310, the mixdown agent 614 transmits the output stream. Theoutput stream can be transmitted to the streaming service 611 or to theclient 633. This can be implemented as a module designed to move theoutput to a specified location (e.g., to a network location or to sendto a device of a user/entity).

FIGS. 14A-14E are diagrams illustrating examples of two audio items thatare processed by the mixdown agent 614. While one of skill in the artwould understand that multiple audio items (or files) can be processedby agent 614, for purposes of clarity and explanation, only two itemswill be discussed herein; however, one of skill in the art wouldrecognize that the disclosed functionality can be implemented on anynumber of files/items without departing from the scope of the instantdisclosure.

FIG. 14A shows a first audio item 1401 and a second audio item 1402.These audio items 1401, 1402 can be obtained by accessing an audiolibrary 622 or by extracting them from an audio stream. The mixdownagent 614 can receive the audio items from the audio library 612 or fromthe streaming service 611.

The audio items 1401, 1402 can be formatted as audio files. Theyrepresent two audio items that are scheduled to be played back inconsecutive order. For example, they can represent consecutive songs ona playlist dynamically generated by a streaming service 611. In someembodiments, the items 1401,1402 can be audio files that are stored in a“smart folder,” as discussed below.

The mixdown agent 614 processes each audio item 1401, 1402 to generate ahead portion 1405, 1414, a middle portion 1408, 1417, and a tail portion1411, 1421. The head portion 1405, 1414 represents the beginning of theaudio item 1401, 1402 while the tail portion 1411, 1421 represents theend of the audio item 1401, 1402. In some embodiments, the middleportion 1408, 1417 is positioned in the middle of the audio item andrepresents a majority of the audio item in terms of length or size.

These portions described above can be determined using AI, machinelearning, DSP, or a wide variety of other algorithmic techniques. Forexample, a CNN model(s) can be implemented to perform the agent 614processing.

In some embodiments, these portions represent moments in the audio item1401, 1402 having significant melodic changes such as, for example, theintro or outro to a song. The transition from the head portion 1405,1414 to the middle portion 1408, 1417 can represent where a song's introends and where vocals or one or more other melodic or significantcomponents of a song begin. The transition from the middle portion 1408,1417 to the tail portion 1411, 1421 can represent where the melody orvocals end and an outro begins in some embodiments.

In some embodiments, the transitions described above can be identifiedby analyzing the zero-crossings of a wave form and/or areas where audioamplitude is relatively low between to zero-crossings. This techniqueenables subsequent processing to reduce the occurrence of an audiblediscontinuity (e.g., perceived as a pop effect, audible “click” or othersonic artifact). In some embodiments, each portion of the audio file canbe formatted as a separate file that is capable of being independentlyprocessed. In some embodiments, these separate files can be rendered(e.g., for example, only audio portions of audio files can be part of anoutput stream, as discussed herein).

In FIG. 14B, the tail portion 1411 of the first audio item 1401 and thehead portion 1414 of the second audio item 1414 are selected forprocessing. The mixdown agent 614 performs a mixdown process 1432 togenerate a mixed down tail portion 1435 and mixed down head portion1437. The mixdown process 1432 can operate according to specifiedparameters obtained as part of the mixdown agent's 114 operation. Thiscan include the operations described above in relation to FIG. 13 .

For example, the mixdown process 1432 can overlap (partially orcompletely), crossfade, or otherwise mix a portion of the tail portion1411 and head portion 1414. The result are two separate audio filesbeing the mixed down tail portion 1435 and mixed down head portion 1437.The mixed down versions can be shorter than the versions prior to themixdown process. For example, in some embodiments, the mixed down tailportion 1435 can be shorter than the tail portion 1411. In addition, themixed down tail portion 1435 can include some content from the headportion 1414. Some embodiments apply smart limiting between the one ormore files of one or more file types. This enables the avoidance ofunwanted audio artifacts caused by overflow conditions when multiple(e.g., two) digital signals are mixed together. In some embodiments,this involves a “brickwall” limiter that can store its envelope followerstate so that it can be re-initialized to avoid audio discontinuities orother effects between any desired portions including, withoutlimitation, the middle and tail in a subsequent mixdown.

According to some embodiments, the mixdown process adds overlay content628 that is mixed into one or both of the mixed down tail portion 1435and mixed down head portion 1437. According to some embodiments, anentire file can be utilized without the portioning discussed herein, asone of skill in the art would understand from the instant disclosure.The overlay content 628 can be identified by the streaming service 611such that it is tailored or otherwise targeted to the user. In thisrespect, the mixdown agent 614 enables customized overlay content to bepresented to a user as one audio (or other file type) item 1401seamlessly transitions into a consecutive audio (or other file type)item 1402

FIG. 14C shows the first audio item 1401 and second audio item 1402being updated to include the mixed down tail portion 1435 and mixed downhead portion 1437. The mixdown agent 614 can stitch or otherwise combinethe separate audio file portions to generate an updated first audio item1401 and updated second audio item 1402.

FIG. 14D shows the updated first audio item 1401 and updated secondaudio item 1402 combined into an output file 1445. The output file 1445includes the content of the first audio item 1401 and second audio item1402 so that they can be played back consecutively in a seamless manner.

According to some embodiments, the larger portions of the tracks (e.g.,the middles 1408,1417) can be pre-processed and pre-encoded. In suchembodiments, when doing the mixdowns, only the heads and tails ofcontent are to be considered (e.g., for mixing them together, addingcontent, slicing up, and then encoding the output (to AAC, MP3, and thelike)). In some embodiments, the delivery of the mixdowns, therefore,can involve transmission to a fileserver (or CDN) that delivers thefiles for gapless reassembly at the client-end.

FIG. 14E shows a diagram illustrating an example of audio items that areprocessed by the mixdown agent 614 according to some embodiments. Someembodiments comprise a mixer H2. Some embodiments comprise overlays ofDJ content, as discussed above. Some embodiments include elements thatcan be blended. In some embodiments, volume can be leveled. Someembodiments comprise one or more skip stops, as discussed below. Asdiscussed below, skip stops can be located anywhere within an audiofile, as they provide for an interactive experience where users can movearound an audio timeline.

FIG. 15 is a diagram illustrating an example of an output file accordingto some embodiments of the present disclosure. FIG. 15 depicts an audiooutput file 1502. The output file 1502 can be any audio output fileprocessed by the mixdown agent such as the output file 1445 described inFIG. 14D. According to embodiments, the mixdown agent 614 is configuredto insert one or more skip stops 1506 between the mixed down tailportion 1435 and mixed down head portion 1437. In some embodiments, theskip stop 1506 can be provided as a separate file including a timestampor it can be included as metadata in the output file 1502. In someembodiments, the data/metadata related to a skip stop 1506 can beappended to a file, or included as header information.

The skip stop 1506 indicates a time position in the output file 1502that should be indicative of a beginning point in the event a skipcommand is received prior to the skip stop 1506. In the event a skipcommand is received, the system can navigate to the skip stop and resumeplayback. In some embodiments, the original head portion 1414 is playedinstead of the mixed down head portion 1437 when navigating to the skipstop 1506. In these embodiments, the user will experience listening tothe second audio item 1402 as if there was no mixdown process 1432.

FIG. 15 also shows an embodiment of the mixdown agent 614 insertingmetadata 1511 into the output file 1502. Metadata 1511 (e.g., 1511 aand/or 1511 b) can include the artist, title, or other information aboutthe audio item. The first audio item 1401 can correspond to firstmetadata 1511 a and the second audio item can correspond to secondmetadata 1511 b. When the audio player 637 of the client device 633plays back the output file 1502, it can recognize and process themetadata 1511. In response, the client device 633 can render for displaythe contents reflected in the metadata.

According to some embodiments, the mixdown agent 614 can implement asingle channel gapless technique where a first and second audio item1401, 1402 can be selected, and then where the first audio item 1401should overlap the second audio item 1402 can be selected as well. Insome embodiments, the head portion and a tail portion can be re-renderedfor each transition and/or content moment.

In some embodiments, the mixdown agent 614 is configured to receive afirst audio item 1401, a second audio item 1402, and a plurality ofoverlay content items. Overlay content items can include a channel forsound effects, audio from an interview, a radio advertisement, and othersources of audio content. The mixdown agent 614 can combine one or moreof the plurality of overlay content items into a single file. Thissingle file can be reused when mixing down different audio items.Additionally, overlay content libraries can be generated and efficientlyused in large numbers of subsequent applications. Some embodimentsprovide functionality similar to a “render form” where a cloud-enabledservice is scaled to create new audio (mixdowns) at virtually anydesired scale.

Turning back to FIG. 5 , having configured and managed the audio file(and other files) (e.g., Step 508), Process 500 turns to the performanceof generating experiences. Step 510.

According to some embodiments, the processing of Step 508 can beperformed via the systems and methods discussed in relation to FIGS. 16and 17 . According to some embodiments, Step 508 can involve thegeneration of playlists, broadcasts, stations, and the like, asdiscussed in relation to FIG. 18 , which is an embodiment of a detailedflow of embodiments discussed in relation to FIGS. 16 and 17 .

Turning to the embodiments of the systems and methods disclosed inreference to FIGS. 16-17 , some embodiments provide for using, amongother variable, content attributes, historical decisions, userattributes and preferences, contextual attributes and dynamicdescriptions of outcome possibilities to make decisions and produceaudio or other content moments and experiences.

Some embodiments dynamically generate high-level instructions thatdescribe how to produce an audio experience, which may be a singleexperience (that can be short or long) or can describe how to produce asuccession of experiences that are chained together. Some embodimentsmake song mixes, podcasts, advertisements, and/or other content asdesired. Some embodiments enable production of one or more lengthy (orpractically perpetual) audio or video experiences through dynamicquerying of databases that can include producer and/or user preferencesregarding a large number of attributes and subjects, followed by dynamicscripting of content completely or substantially consistent with thepreferences. In some embodiments, such preferences can be adjusted fordifferent experiences, producers, users and branding goals.

High quality content generation typically requires significant manualinputs from skilled personnel. As just one non-limiting example, audioproduction services allow a client device to select a digital stationand receive an encoded audio stream that the client device can decodeand playback via one or more speakers. Audio production services mayinclude a server-based application that selects different audio filesfrom a library and transmits them for playback in serial order. Priorart systems lack the ability to dynamically query and script contentexperiences to produce a desirable experience, and are not designed forpersonalized or contextualized delivery on today's content services

Some embodiments use an intelligent and dynamic querying and scriptingengine to assemble and generate either directives for stand-aloneexperiences (e.g., podcast, ad) or directives for the contextualinsertion or overlay of content between/on two songs (or other contentelements.) Some other embodiments use an intelligent and dynamicquerying and scripting engine to assemble and content such as playlists,albums, advertisements, podcasts or other related or standalone contentas well as content to be inserted between audio events, such as, forexample a transition between two songs.

The content generator 615 is utilized for the performance of the systemsand methods of FIGS. 16-17 , as illustrated in FIGS. 6 and 10 .

As illustrated in FIG. 6 , streaming service 611 can function as aproduction service (referred to herein as “production service” 611 forpurposes of the description of FIGS. 16-17 ). The production service 611and content generator 615 mentioned above are components executed on thecomputing system 601. These components may generate data and store thedata on the database 603 and/or access the contents of the database 603.The production service 611 may be an application implemented on one ormore webservers that enable users to subscribe, create, edit, and managestandalone or sequences of content. In some embodiments, the productionservice 611 receives user input and generates an encoded audio streamthat is transmitted over the network 616 for playback.

In some embodiments, the content generator 615 can produce podcasts,create audio tracks for videos, create advertisements for playback on awide variety of platforms, create music playlists and associatedcontent, and the like. Some embodiments provide standalone contentcontextualized for a personalized and desirable experience.

The content generator 615 can comprise a software application or modulesthat communicate with the production service 611. The content generator615 may employ one or more APIs to plug into the production service,receive control commands and data from the production service 611 andgenerate output data that is transmitted to the production service 611.

In some embodiments, the data stored in the database 603 includes anaudio library 622. In some embodiments, as mentioned above, the library622 can be partitioned into, or include portions (e.g., structures) of aprimary content library and a secondary content library. In someembodiments, asset features, as discussed below, can be housed/stored inlibrary 622 (in either primary and/or secondary libraries) or storeseparately in a portion of database 603.

According to some embodiments, the primary content library may be alibrary of audio files that a user may wish to stream. The primarycontent library may comprise, among other forms/types of data/metadata,several song files, music files, podcasts, or other relatively longaudio files that make up substantive content for entertainment purposes.The secondary content library is a series of clips or pre-recordingsthat may be informative or support the presentation of information. Thismay include, for example, audio clips announcing the radio station,advertisements, informative recordings, sound effects, and backgroundmusic, and the like.

In some embodiments, the audio library 622, inclusive of the primarycontent library and secondary content library, may be embodied as a setof databases with features that describe audio files and/or audioportions of video files, e.g., representations of the files themselves.A library may include metadata such as artist, title, album information,chapter information, and the like.

In some embodiments, asset features include data generated fromanalyzing the contents of the primary content library and secondarycontent library. In some embodiments, asset features are generated usingmachine learning or other artificial intelligence algorithms. The assetfeatures may indicate information about an audio item such as the key ofthe music, the chords at the beginning and ends of songs, the degreethat two pieces of audio are in tune, the energy level of an audio item,or any other attribute or quality about an audio or video item. In someembodiments, the asset features can also include metadata that is addedat the time of import or any other point thereafter by humans or othersources or processes. In some embodiments, this also can include thetext of the content (if speech) as extracted by ML/DSP processes or anyother data extracted or produced by selected sources or processes, asdiscussed above.

In some embodiments, the ML/DSP processing can involve, but is notlimited to, performing source separation (if/when determined necessary)to isolate the human voice component—then this portion is analyzed by aspeech to text engine/model. This allows the identification of the textof content that may have music and other elements mixed in with it(e.g., an advertisement). In some embodiments, the text providesassistance for sentiment analysis as well as basic search indexing. Insome embodiments, the human voice component may include singing whichcan be analyzed by a speech to text engine/model to obtain lyrics.

According to some embodiments, the content generator 615 is configuredto enhance the functionality provided by the production service 611. Thecontent generator 615 may receive the audio items selected by theproduction service 611. In addition, the content generator 615 mayobtain audio items from the secondary content library 625. In someembodiments, production service 611 may provide conditions on which toconfigure the operation of the content generator.

In some embodiments, content generator 615 identifies one or more audioitems from the secondary content library, assembles them as desiredincluding overlapping, interaction, and any other desired effects orresults, and inserts the assembled content.

In some embodiments, one or more concordance rules are used once ormultiple times. One non-limiting example of a concordance rule is whenthe system detects an ad signal and an upsell opportunity, the systemcombines two moments together. In some embodiments, this may beprocessed by a “combinational formulae”, which can be generalized formore than one formulae, or for specifically identified formulaescenarios (e.g., ads and upsells, for example).

In some embodiments, the content generator 615 can access a database ofrules (which may range from simple to complex) and deliver contentrotating through rules and/or formulae (or formulas, usedinterchangeably) as desired. For example, formulae can indicate where toinsert (or “stitch-in” via a mixdown, as discussed above) anadvertisement, branding audio or other voice-over content. For example,a formula may indicate that at predetermined times or intervals,additional content is to be “stitched” in, as discussed herein in moredetail.

In some embodiments, a formula is made up of one or more elements. Theseelements can directly reference a particular piece of content, but thisis less common in some embodiments. In many embodiments, they representa dynamic query for content. For example, a query can be executed for avoiceover liner that is appropriate to a particular listening contextand within a certain margin of the incoming song's energy level. In someembodiments, a formulae engine retrieves all of the content that matchesthat criteria and further evaluates it to pick the best one; e.g., theone heard least recently, that is closest in energy level, and that fitsbest over the introduction of the incoming song. This way, in someembodiments, the dynamic formula retrieves and ranks different contentbased on the time and context that it is being executed in.

In some embodiments, the rules database can comprise any databaseincluding, without limitation, an object-oriented database which can bedynamically queried. In some embodiments, dynamic querying and scriptingcan provide a highly personalized experience for users. Some embodimentsof formatics are flexible and can be optimized using feedback fromvarious conventional sources. Some embodiments comprise runtime criteriaand control how content events are dispensed over time.

Some embodiments first schedule content using formulae which have firstbeen tested to see if they work well. In some embodiments, thescheduling is enabled working down or up a list of formulae. Next, insome embodiments, content is dispensed out over time for given users andtheir respective histories. Next, in some embodiments, directives aretranslated, yielding instructions such as playlist ordering or otherdesired content sequencing.

Some embodiments enable highly customized content treatment usingformatics to tweak factors and weigh relevant evidence or desiredcharacteristics. In some embodiments, the weighting can also account forsurrounding content and/or discrete audio elements, which can be used inrendering the formulae. Some non-limiting examples include being moreaggressive regarding voiceover or other content generation,modifications regarding acceptable relative amplitude between adjacentor overlaid content, and recombining sources with softer vocals or otherelements. Some embodiments use rotating rules such as an ad injection orbranding event as predetermined periods of time elapse.

Some embodiments provide a system for generating and managing audiomoments, wherein new software code is not needed for each new conceptfor desired effect. Some embodiments provide the flexibility andcapability to create virtually any desired content, in ways that are notdogmatic and can take the place of human input if desired. Someembodiments provide the ability to automatically generate completecontent for radio or other media stations or outlets using predeterminedand/or flexible formulae to provide a great user experience.

FIG. 16 illustrates a non-limiting example of the content generator 615according to some embodiments of the present disclosure. FIG. 16 depictsthe audio analyzer 1605, formulae engine 1608, audio assembler 1611 andscheduler 1614. Each of these components may be separate modules thatmake up the content generator 615. In some embodiments, these modulesmay be embedded into the production service 611.

In some embodiments, the audio assembler 1608 transitions from one audioitem to another audio item. The audio assembler 1608 may generate theplaylist of primary audio content or receive the playlist from theproduction service. In some embodiments, the audio analyzer 1605performs feature extraction and classifies asset features to describeaspects of audio items. The audio analyzer 1605 may be configured tooperate on any content such as, for example, content from the primarycontent library and from the secondary content library of audio library625 (as discussed above). In some embodiments, the audio analyzer 205generates asset features and stores them in a database 603.

In some embodiments, the scheduler 1614 identifies different conditionsto drive the selection of one or more formulae. In some embodiments, aformula may be a data structure that is dynamically generated from ascript. In some embodiments, a formula may comprise a set of rules orexecutable instructions providing information and control as to how togenerate content.

Non-limiting examples of formulae include an interstitial formula, anupsell formula, a back-sell formula, an advertisement formula, or anyother formulae for arranging a sequence of audio files. In someembodiments, an interstitial formulae may provide information about alistening context, or a name of the listening context. In someembodiments, an interstitial formula may include a combination of amusic embedded effects and an audio clip stating the listening context'sname. A listening context can be, but is not limited to, a radiostation, playlist, a streaming service, content channel, area of aservice, or other organizing factor that is used to differentiate anarea where a set of behaviors apply.

In some embodiments, an upsell formula may include an introduction clipand a subsequent clip stating the artist's name. This is referred to as“concatenation”, and it uses not only the intent of the pieces (and theindividual pieces' relationships with the content around it), but alsoan analysis of speech rhythm and cadence to ensure that the timings,amplitudes, and vocal inflections for the elements are well-matched andobserved. An introduction clip may be an audio recording of a voicesaying “up next is”. A back sell formula may include a summary clipfollowed by a clip stating the artist's name. A summary clip may be anaudio recording of a voice saying, “you just listened to”. Anadvertisement formula may comprise one or more clips for presenting anadvertisement.

In some embodiments, scheduler 1614 selects a particular formula orformulae based on conditions. In some embodiments, conditions mayindicate when to generate content based on the secondary content libraryof library 625 and what kind of content to generate. The productionservice 611 may provide specific conditions to the scheduler 1614. Forexample, the condition may indicate that the content generator 615should generate content towards the end of a particular song and thecontent should be an advertisement.

In some embodiments, once the scheduler 1614 selects a formula orformulae based on the conditions, the formulae engine 1608 identifieswhat audio items to play and in what order to play them, where suchclips are taken from the secondary content library of library 625. Insome embodiments, the formulae engine 1608 may select clips based on theasset features of audio items. In some embodiments, the formulae engine1608 may analyze metadata or tags associated with audio items to alsoobtain information about the audio item.

In some embodiments, using metadata and/or asset features, the formulaeengine 1608 identifies one or more audio clips from a secondary contentlibrary of library 625 that best matches or is consistent with the twoaudio items. This functionality is described in further detail withrespect to FIG. 3 .

In some embodiments, the formulae engine 1608 orders the audio itemsthat are selected from the secondary content library of library 625. Insome embodiments, the audio items may partially or fully overlap withone another. For example, a background music clip may play overlap withan advertisement clip made up of pure vocals.

Once the formulae engine 1608 identifies the audio items from thesecondary content library of library 625 and orders it, the audioassembler 1608 combines the audio items and inserts them at a timeposition that coincides with the occurrence of an audio event.

In some embodiments, the formulae engine 1608 can process and determinemultiple types of moments: start, end, interlineal, overlay, andstandalone. Formulae engine 1608, therefore enables the contentselection and production, as discussed herein and below.

FIG. 17 illustrates an example of audio data processed according to someembodiments of the present disclosure. Specifically, FIG. 17 shows howthe content generator 615 may generate content that is inserted into astream of content taken from the primary content library of library 625.The example depicted in FIG. 17 is non-limiting, as any number of songs,clips and the like can be interlaced according to the disclosedtechniques and mechanisms of the content generator 615.

For example, as illustrated in FIG. 17 , a first song 1705 is to beplayed followed by a second song 1706. Both songs 1705, 1706 may beaudio files taken from the primary content library of library 625. Thesesongs 1705, 1706 may be played or streamed by the production service611. The system determines that the transition from the first song 1705to the second song 1706 is an audio event is a condition that could usedynamically generated content. In some embodiments, the contentgenerator 615 is instructed to generate content dynamically to beinserted at a time position around this transition event. In the exampleof FIG. 17 , the condition is for upselling the artist associated withthe second song 1706.

In some embodiments, based upon these conditions, the scheduler 1614selects an upsell formulae from a list of predetermined formulae. Theformulae engine 1608 is instructed to generate content according tothese conditions. In some embodiments, based on one or more assetfeatures associated with the first song 1705 and/or second song 1706,the formulae engine 1608 identifies a background music clip 1708 fromthe secondary content library of library 625. For example, the assetfeatures may indicate the first song 1705 and/or second song 1706 fallwithin the jazz musical genre. Accordingly, the formulae engine 1608selects a background music clip 1708 that is labeled or tagged as jazz.

In some embodiments, the formulae engine 1608 then selects an intro clip1711. The selected intro clip 1711 may be purely voice with no musicsuch that it is musically compatible with mixing it with the backgroundmusic clip 1708. In addition, the intro clip 1711 may be selected fromone of a plurality of intro clips. The selected intro clip 1711 maycorrespond to the jazz genre, or match the tempo, pace, or energy levelof the first or second song 1705, 1706.

In some embodiments, the formulae engine 1608 then selects an artistclip 1714, which may be a vocal recording of a person saying theartist's name. In this case, the formulae engine 1608 may accessmetadata associated with the second song 1706 to determine the artistand then identify the appropriate artist clip 1714.

In some embodiments, the audio assembler 1611 may then combine thebackground music clip 1708, intro clip 1711, and artist clip 1714, eachof which was selected from the secondary content library. In someembodiments, these audio items may be combined so that the backgroundmusic clip 1708 overlaps with the intro clip 1711 and artist clip 1714.In addition, in some embodiments, the intro clip 1711 is positionedimmediately before the artist clip 1714 to create a seamless transition.In some embodiments, content generated from the background music clip1708, intro clip 1711, and artist clip 1714 is then inserted at theaudio event, which is the transition from the first song 1705 to thesecond song 1706. In some embodiments, the generated content (e.g., thecombination of background music clip 1708, intro clip 1711, and artistclip 1714) overlaps at least partially with the end of the first song1705 and/or beginning of the second song 1706.

When stitching together songs and clips, some embodiments are directedto identifying time positions for where two audio files should overlapto improve the user's music listening experience. For example, an audiorecording for an advertisement selected from the secondary contentlibrary of library 625 should overlap with a song selected from aprimary content library of library 625 in a manner that does notinterfere with the listening experience. For example, the audio of anadvertisement should not interfere with the vocals of a song.Furthermore, the audio advertisement can end immediately before amusical moment (e.g., introduction of vocals, introduction of aninstrument, and the like) of a song begins. This can create musicalcontinuity that improves the listening experience.

In some embodiments, a musical moment can be determined usingcomputational musicology. For example, a song's waveform or frequencytransformation may be analyzed to identify its beat structure, frequencysignatures, instrument entry points, or vocal entry points, and thelike. A song's waveform being analyzed can be divided into partsincluding, for example, individual instruments, drums, vocals, or anyother component. The beat of a song can also be determined to identifythe transitions between measures. A DSP/ML, can be used to identify suchmusical moments. These musical moments can be used to identify points ofoverlap when stitching content together.

In some embodiments, trained machine learning models, such as CNNs, forexample, can be used to determine musical movements, as discussed above.In this case, features in a song can be identified or extracted and thenlabeled to create training data. The training data may be used to traina classifier to identify moments in a song where overlays or otheradditional content or effects are permitted and where changes are notpermitted.

In some embodiments, the scheduler 1614 can implement a series of rulesfor generating content made up of audio from the primary content libraryand secondary content library of library 625. These rules may be used tocreate variety and avoid repetition when dynamically generating contentto be played by a production service 611. The secondary content libraryof library 625 may comprise a plurality of audio clips and phrases.Voice talent can manually record such audio clips. The content of a clipcan be recorded several times to correspond to different energy levels,intensities. Several recordings can be made using different words toconvey a similar message. This creates a library comprising variation tomake the dynamically generated content sound less mechanical or moreorganic. For example, one clip can say “up next is” while another clipcan say “next is”. In addition, these clips may be recorded by differentpeople and/or with different inflections and/or different energy levels.In some embodiments, it can be specified and/or determined whethervoices, for example (or other characteristics) of the particularfragments match or do not match.

In some embodiments, scheduler 1614 can leverage play history (for asingle user, group of users, or content channel/station) via rules thatdistribute individual content items across time. This way, the sameaudio is not being consistently rendered, thereby creating a redundantlistening experience. In some embodiments, content can be distributed aspart of a grouping for a predetermined period of time, for particularusers, for particular contexts, and the like; therefore groups ofcontent can be played together in instances where deemed to fit theschedule; however, rules exist which prohibit their constant renderingthereby avoiding overplay which can lead to user exhaustion.

In some embodiments, the audio clips of the secondary content library oflibrary 625 may be compressed or stretched while preserving pitch toobtain time variations among each clip. For example, a single recordingof a person saying, “You are listening to classic rock radio” that last5 seconds may be compressed to be 4 seconds or stretched to be 6 secondswhile preserving the pitch. Thus, one clip can be replicated intoseveral clips with varying audio qualities.

When implementing rules, the scheduler 1611 may use opportunistic rulesand/or rotational rules. Opportunistic rules can focus on frequentlychecking if a particular clip or category of clips can be used whengenerating dynamic content. Rotational rules operate according to asequence of clips to try and operate according to additional rules foradvancing through an ordered list.

In some embodiments, the scheduler operates according to an orderedchecklist to play a particular clip or type of clip. If there is noopportunity to play a particular clip or type of clip, it continues tothe next item on the checklist until a clip or type of clip can beplayed. The scheduler can check off a played clip or type of clip andthen proceeds to the top of the list at the next opportunity.

As an example, in some embodiments, a user can listen to a contentchannel called “Artist A” where the channel plays songs relating to amusical artist referred to as “A”. A content channel can be, forexample, a streaming service station. This channel setting can bereferred to as a listening context, which can be defined by one or moredescriptors (e.g., name and identifier) that uniquely identifies theunique location of a listening experience within a broader hierarchy oflistening experiences. Depending on the listening context, the scheduler1614 can operate according to rules to identify the next clip or clipsto use when dynamically generating content. The rules can specify anenergy level, clip length, a formula, a memory parameter, or any otherparameter.

In some embodiments, regarding the energy level, each clip can be taggedto reflect the clips' energy level. Thus, a clip type may be aparticular energy level. The clip length can refer to the duration ofthe clip. For example, depending on the musical moments of a first song1705 (e.g., outgoing song) and the second song 1706 (e.g., incomingsong) for which the clip is to be inserted, the duration can vary. Thefirst song 1705 can have a long outro making it a good candidate foroverlaying a longer clip over the outro. The memory parameter can referto how long the system should wait before playing a particular clip ortype of clip. In some embodiments, memory parameter can refer to howlong a user's play history is saved.

In some embodiments, the rules can limit the dynamic creation ofupselling content to improve the listening experience. Or, in someembodiments, an upselling formula can take priority to play morefrequently if an administrator of the system desires.

In some embodiments, rules can be used to vary the content selected fromthe secondary content library of library 625. In some embodiments, rulescan be used to prevent the repetition of certain types of clips for aspecified period of time (e.g., using a memory parameter). In someembodiments, rules can be used to prioritize certain types of clips overothers or play a particular clip whenever an opportunity presentsitself.

Some embodiments provide directives or other input to inform playbackeither directly or being passed through a second service that providesmore precise timings and other relevant information if desired. Onenon-limiting example of a second service that can work synergisticallywith the innovations described herein is described in U.S. Pat. No.10,409,546, the content of which is incorporated herein in its entirety.Such directives or other information can be rendered at one or moreunicast playback clients or via a cloud agent in a broadcast studio forterrestrial satellite, or internet Multicast delivery. One non-limitingexample of a second service that can work synergistically with theinnovations described herein is described in U.S. Pat. No. 10,509,622,the content of which is incorporated herein in its entirety.

Turning to FIG. 18 , Process 1800 details embodiments of steps performedby engine 400, including executed logic performed at least by contentgenerator 615, for providing renderable content. Process 1800 providesnon-limiting example embodiments of the disclosed framework's operationupon the uploading, storing and cataloging of audio files (e.g.,performed via the preceding figures). That is, the operations of Process1800 operate in-line with creating a renderable audio experience for auser or set of users.

In some embodiments, a user can be a third party content provider thatis requesting the creation of a playlist. In some embodiments, therequest can include content that can be used as a seed file. In someembodiments, the request can include a directive upon which the playlistis created (e.g., information about an audio file, such as a context),as discussed in more detail below. In some embodiments, the user can bea user of a third party platform, or a user that is a subscriber to suchplatform or a subscriber to a service that is provided by the disclosedframework.

According to some embodiments, Process 1800 involves the creation of aplaylist as a renderable audio experience for a user in response to arequest from a third party. In Step 1802, the request comprisesinformation related to an audio file. In some embodiments, theinformation can include, but is not limited to, a name of the audiofile, an identifier for the audio file, the actual audio file, a pointeror network address of the audio file (e.g., a uniform resource locator(URL)), name of an artist, name of a genre, a feeling, a mood, or otherform of emotion or feeling that is captured by a piece of music, alength, a time period, a context (e.g., what is the song about, what isthe song referencing), at least a portion of the lyrics, an album name,album cover art and the like, or some combination thereof.

In some embodiments, Process 1800 can be applied to a situation wherethe request comprises a set of audio files, which are to be arranged atthe discretion of engine 400—this is performed in a similar manner asdiscussed herein, where the audio files included in, or referenced inthe request, are leveraged in a similar manner as discussed below inrelation to the identified audio file from Step 1802. Thus, one of skillin the art would recognize that the quantity of audio files received atthe onset of Process 1800 would not change the scope or functionality,as engine 400 is capable of creating a user experience based on seedcontent, where the content can reference a single audio file or a feedof audio files (e.g., a playlist).

In some embodiments, the request in Step 1802 can further includeinformation related to a setting for rendering the audio file and itssubsequently identified audio files, whether they are voice overs,advertisements, or other songs. Such information can include, but is notlimited to, asset features and conditions that drive the formulae engine1608 and scheduler 1614, as discussed above. For example, the requestcan reference that it is December, and that holiday-type music isrequested. Rather than play Christmas-music, for example, the requestordesires upbeat, “happy” music—therefore, in this example, the requestcan include information indicating an energy level value and/or aminimum threshold for BPM, with lyrics having a “context” thatcorresponds to a “happy” mood.

In some embodiments, these settings can be provided or applieddynamically, either as the playlist is being compiled and/or as it isrendered, as discussed in more detail below (and illustrated via thefeedback loops/lines in FIG. 18 ).

In Step 1804, upon receiving the request, engine 400 analyzes therequest to identify the information which will form the seed upon whicha playlist is compiled. In some embodiments, when the request eitherincludes the seed audio file, or directly references it, Step 1804involves parsing the request and identifying the relevant informationrelated to the seed file.

In some embodiments, the request may include information referencing anaudio file (e.g., metadata related to a song, as mentioned above). Insuch embodiments, Step 1804 can include analyzing the contextualinformation included therein by any known or to be known analysistechnique, algorithm, classifier or mechanism, including, but notlimited to, ANNs, CNNs and computer vision, for example. This analysisenables engine 400 to determine or identify which audio file is beingreferenced.

In Step 1806, the attributes of the audio file identified in Steps1802-1804 are identified. The identification of these attributes can beperformed using any of the above techniques discussed in relation to atleast Step 504 of FIG. 5 .

In some embodiments, Step 1806 results in the identification ofattributes, which include, but are not limited to, melodic features,tempo regions, amplitudes, beats per minute (BPM), fade ins/outs,features of individual stems (using source separation), dominantfrequency ranges, structure, beat positions, onsets, harmonics,speakers/singer quantity, background noise, energy level, pitch, silencerates, duration, sonic genre classification (multiple classificationswith or without weights), loudness, key, meter, gender of vocals (maleor female), arrangement (music with vocal or instrumental), mood(happiness and sadness), character (acousticness and electronicness),danceability, harmony (tonal or atonal), attitude (aggressiveness andchillness), environmentalness (music or environmental sounds),environmental sonic genre (multiple classifications with or withoutweights), and/or any other acoustic or DSP metric, value orcharacteristic that is identifiable from an audio file, or somecombination thereof, can be determined, derived, extracted or otherwiseidentified.

In Step 1808, engine 400 formulates a search query that is utilized tosearch at least one of the databases discussed above. As mentionedabove, these databases include audio information stored as content, keyvalue pairs, feature vectors, and the like. In some embodiment, asdiscussed above, the databases can function as a multidimensionaldatabase(s) that comprises n-dimensional layered data related tospecifically formatted and stored audio data and metadata. As mentionedabove, in some embodiments, particular types of audio content and/orattributes/features are stored in particular databases.

Therefore, for example, the formulated query in Step 1804 can includestring or sequence of queries (e.g., a multidimensional query) that areto be executed in parallel. For example, the deep features of the audiofile (e.g., harmonics, and the like) can be translated into a featurevector for performing a search of a feature database, as performed in asimilar manner as discussed above. Additionally, or in the alternative,information related to the context from the audio portion (e.g.,text-to-speech) can be subject to Natural Language Processing (NLP)techniques, and used to query a content database as a text string.

In some embodiments, the query formulated in Step 1808 can includecriteria for guiding the search, such as, but not limited to, a numberof audio files to be identified during the search, a type of audio file,a ratio of particular types of files or content within files, and thelike.

For example, rather than just returning a number of songs (e.g., 25songs, for example), the query can request 13 songs, and 12 voice overs,so as to create a music experience, rather than simply a streamedstation, as in conventional systems. As mentioned above in relation toat least FIG. 16 , the voice over files can include, but are not limitedto, advertisements, up-sells, back-sells, interstitials, and the like.

In some embodiments, the query can also include information as to a typeof mixdown—for example, which types of transitions, and how long oftransitions between files, can impact types of content and/or files thatare discovered and/or which databases are searched.

In some embodiments, the query can also include information related toformulae, as discussed above in relation to the implementation offormulae engine 1608 of FIGS. 16-17 .

In Step 1810, engine 400 executes a search(es) based on the formulatedqueries. As mentioned above, the queries are performed on the associateddatabases that are connected (e.g., either remotely (e.g., in the cloud)or locally (e.g., server information)) to the hosting platformproviding/executing engine 400.

In some embodiments, results from certain databases are used toautomatically query another database for similar information. Forexample, upon performing a query of the feature database, a result of acluster of audio information is identified; however, this informationmay be compiled as vector information for the audio cluster. Forexample, audio files with node-features on a respective vector beingwithin a threshold Euclidian distance. This vector information is thenleveraged as another query of a content database, for example, as a setof key-value pairs for each item in the cluster, to identify each of theaudio files referenced in the cluster that are housed in the digestincluded in the content database.

In Step 1812, the results are identified, analyzed and a schedule (e.g.,a playback data structure) for each file in the search results and theaudio file identified in the request (from Step 1802) is determined.According to some embodiments, Step 1812 involves receiving the compiledresults, analyzing them (e.g., via scheduler 1614) and determining anorder of each audio file identified in the search results, as well asany overlap, if any, between transitions of files while maintaining agapless interplay between each file (e.g., via mixdown agent 614, asdiscussed above in relation to FIGS. 13-15 ).

An example of this is discussed above and illustrated in FIG. 17 . Insome embodiments, as illustrated in FIG. 17 , the order (or sequence)corresponds to when one audio file begins playing respective to whenanother audio file beings playing (e.g., if one begins playing beforeanother, yet there is overlap with another file), then that file issequenced prior to the other file in the schedule/order. In someembodiments, if two files begin playing at the same time, then thelength of the file will dictate which is ordered first (e.g., theshorter file is first since its rendering will finish first).

In some embodiments, as discussed above in relation to FIGS. 13-17 ,scheduler 1614 of content generator 615, as well as mixdown agent 614,can execute any known or to be known type of analysis, scheduling andaudio blending technique, including, but not limited to, ANNs, CNNs andcomputer vision (e.g., computer audition).

In Step 1814, the information resultant of the compiled schedule andmixdowns from Step 1812 is stored. The storage is performed in a similarmanner as discussed above in relation to at least Step 1808 whereappropriate data (e.g., vector data, key-value pairs, context data, forexample) is stored in a corresponding database associated with engine400.

In Step 1816, the AI/machine learning models (e.g., CNNs andclassifiers, for example) implemented by engine 400 (e.g., musicprocessor 613, mixdown agent 614 and content generator 615, as discussedabove) are then trained, or further trained on this information so thatfuture search results, schedules and mixdowns can be performed moreaccurately and computationally efficiently.

As mentioned above, the requesting user can provide input, settings orparameters for controlling how the playlist is managed. For example, avalue of danceability, happiness, energy, or any other musicalityattribute can be set, modified and/or controlled. Thus, in someembodiments, this can involve leveraging these input variables to modifyhow the audio files are rendered or even ordered. In some embodiments,such modification can cause a re-search (e.g., performance of Step 1810again. In some embodiments, energy levels or other parameters can be“sculpted” (modified using a configurable curve on a user interface orother methods) over time to yield the desired listening experience.

In some embodiments, Step 1818 can be performed, which monitors forthese types of inputs by the requesting user. Should input be providedat this stage, Process 1800 would proceed back to at least Step 1810 or1812 to search and/or recompile the schedule and mixdowns. In someembodiments, a re-search can be performed, as mentioned above.

Thus, Step 1818, and the double-lines between Steps 1810 and 1812, and1812 and 1818, as well as the line in the drawing figure from step 1802to Steps 1812 and 1818 illustrate that parameters can be input by therequesting user as the onset of the request for a playlist, during thecompilation operations, or after a playlist is compiled (e.g., as it isplaying, for example), where the playlist can then be modifieddynamically, in-real time.

Thus, in Step 1820, without receiving further input (from Step 1818),the compiled and scheduled playlist of audio files (e.g., an example ofsuch is illustrated in FIG. 17 ) is rendered. Such rendering can involvestreaming the audio files over a network. In some embodiments, suchrendering can involve sending a message(s) to the requesting user thatincludes information related to the playlist. In some embodiments, thescheduled playlist can be hosted on an FTP site. In some embodiments,the scheduled playlist can be stored in a “smart folder,” as discussedbelow.

In some embodiments, the playlist can act as a data structure for therequesting user to render the audio from his/her location, or over anetwork. In some embodiments, the playlist can function as a broadcaststation, whereby subscribers can tune-in to hear the playback from adedicated network location. In some embodiments, the playlist can beused for on-demand services, as either a main content portion (e.g., aradio station, news-reel, or podcast), or as background information(e.g., to be played while other content is being visibly streamed orplayed).

In some embodiments, rather than manually creating lists of content ormusic, Process 1800 can be used to populate an interactive, dynamicallyupdateable virtual collection of content meeting the selection criteria,referred to as a “smart folder”. The smart folder acts as a collectingmechanism for audio files that satisfy a user's requested criteria. Forexample, the request in Step 1802 can include information for collectingsongs from an artist X that span 2 minutes to 3 minutes. In anotherexample, a smart folder of voiceover content can be created thatcollects audio that share the same energy level (e.g., high energy),voiceover talent (e.g., Sally and/or Bob), and playback context (e.g.,country music experiences). Process 1800, in a similar manner asdiscussed above, can leverage smart folders for any purpose where acollection of items may be used. For example, when building a playlistor selecting from one or more pieces of content to use for anexperience.

In some embodiments, Step 1802 can include the request providing areference to a smart folder, whereby the playlist is compiled in asimilar manner as discussed above based on the audio files indexed bythe smart folder.

For the purposes of this disclosure a module is a software, hardware, orfirmware (or combinations thereof) system, process or functionality, orcomponent thereof, that performs or facilitates the processes, features,and/or functions described herein (with or without human interaction oraugmentation). A module can include sub-modules. Software components ofa module may be stored on a computer readable medium for execution by aprocessor. Modules may be integral to one or more servers, or be loadedand executed by one or more servers. One or more modules may be groupedinto an engine or an application.

For the purposes of this disclosure the term “user”, “subscriber”“consumer” or “customer” should be understood to refer to a user of anapplication or applications as described herein and/or a consumer ofdata supplied by a data provider. By way of example, and not limitation,the term “user” or “subscriber” can refer to a person who receives dataprovided by the data or service provider over the Internet in a browsersession, or can refer to an automated software application whichreceives the data and stores or processes the data.

Those skilled in the art will recognize that the methods and systems ofthe present disclosure may be implemented in many manners and as suchare not to be limited by the foregoing exemplary embodiments andexamples. In other words, functional elements being performed by singleor multiple components, in various combinations of hardware and softwareor firmware, and individual functions, may be distributed among softwareapplications at either the client level or server level or both. In thisregard, any number of the features of the different embodimentsdescribed herein may be combined into single or multiple embodiments,and alternate embodiments having fewer than, or more than, all of thefeatures described herein are possible.

Functionality may also be, in whole or in part, distributed amongmultiple components, in manners now known or to become known. Thus,myriad software/hardware/firmware combinations are possible in achievingthe functions, features, interfaces and preferences described herein.Moreover, the scope of the present disclosure covers conventionallyknown manners for carrying out the described features and functions andinterfaces, as well as those variations and modifications that may bemade to the hardware or software or firmware components described hereinas would be understood by those skilled in the art now and hereafter.

Furthermore, the embodiments of methods presented and described asflowcharts in this disclosure are provided by way of example in order toprovide a more complete understanding of the technology. The disclosedmethods are not limited to the operations and logical flow presentedherein. Alternative embodiments are contemplated in which the order ofthe various operations is altered and in which sub-operations describedas being part of a larger operation are performed independently

While various embodiments have been described for purposes of thisdisclosure, such embodiments should not be deemed to limit the teachingof this disclosure to those embodiments. Various changes andmodifications may be made to the elements and operations described aboveto obtain a result that remains within the scope of the systems andprocesses described in this disclosure.

What is claimed is:
 1. A method comprising: analyzing, by a computingdevice, an audio file; determining, by the computing device, based onthe analysis, attributes of the audio file, the attributes comprisinginformation related to features of the audio file; analyzing, by thecomputing device, the attributes of the audio file; determining, by thecomputing device, based on the analysis of the attributes, mixinginformation for the audio file, the mixing information identifying aportion of the audio file that is eligible for mixing with another file,the mixing information further identifying an ineligible portion that isineligible for mixing with another file, the ineligible portion being apart of an audible section of the audio file where overlaying additionalaudio is not permitted; generating, by the computing device,instructions for mixing audio data consistent with the mixinginformation; and generating, by the computing device, a stream of audiodata based on the generated instructions.
 2. The method of claim 1,wherein the generated audio data corresponds to at least one of a fileand a stream delivered to a network.
 3. The method of claim 1, whereinthe generated instructions are used to generate a stream of audio datawhich is sequenced to enable hitting the post at the beginning of theineligible portion.
 4. The method of claim 1, wherein the audio datamixing includes a sequence determination that is based on a set offormulae, the formulae comprising information for ordering audio data atpredetermined times or intervals.
 5. The method of claim 2, furthercomprising: receiving input parameters from a user, the input parameterscorresponding to at least some of the features and characteristics ofthe audio file.
 6. The method of claim 2, wherein the stream is a basisfor a broadcast station.
 7. The method of claim 1, further including atleast one multidimensional database that comprises a plurality of datastructures for specific types of the audio features.
 8. The method ofclaim 2, wherein the stream comprises song content and voice-overcontent.
 9. The method of claim 1, wherein the audio file comprisesthird party content.
 10. A non-transitory computer-readable storagemedium tangibly encoded with computer-readable instructions, that whenexecuted by a computing device, perform a method comprising: analyzing,by the computing device, an audio file; determining, by the computingdevice, based on the analysis, attributes of the audio file, theattributes comprising information related to features of the audio file;analyzing, by the computing device, the attributes of the audio file;determining, by the computing device, based on the analysis of theattributes, mixing information for the audio file, the mixinginformation identifying a portion of the audio file that is eligible formixing with another file, the mixing information further identifying anineligible portion that is ineligible for mixing with another file, theineligible portion being a part of an audible section of the audio filewhere overlaying additional audio is not permitted; generating, by thecomputing device, instructions for mixing audio data consistent with themixing information; and generating, by the computing device, a stream ofaudio data based on the generated instructions.
 11. The non-transitorycomputer-readable storage medium of claim 10, wherein the generatedaudio data corresponds to at least one of a file and a stream deliveredto a network.
 12. The non-transitory computer-readable storage medium ofclaim 10, wherein the generated instructions are used to generate astream of audio data which is sequenced to enable hitting the post atthe beginning of the ineligible portion.
 13. The non-transitorycomputer-readable storage medium of claim 10, wherein the audio datamixing includes a sequence determination that is based on a set offormulae, the formulae comprising information for ordering audio data atpredetermined times or intervals.
 14. The non-transitorycomputer-readable storage medium of claim 11, further comprising:receiving input parameters from a user, the input parameterscorresponding to at least some of the features and characteristics ofthe audio file.
 15. The non-transitory computer-readable storage mediumof claim 11, wherein the stream is a basis for a broadcast station. 16.The non-transitory computer-readable storage medium of claim 10, furtherincluding at least one multidimensional database that comprises aplurality of data structures for specific types of the audio features.17. The non-transitory computer-readable storage medium of claim 11,wherein the stream comprises song content and voice-over content. 18.The non-transitory computer-readable storage medium of claim 10, whereinthe audio file comprises third party content.
 19. A device comprising: aprocessor configured to: analyze an audio file; determine, based on theanalysis, attributes of the audio file, the attributes comprisinginformation related to features of the audio file; analyze theattributes of the audio file; determine, based on the analysis of theattributes, mixing information identifying a portion of the audio filethat is eligible for mixing with another file, the mixing informationfurther identifying an ineligible portion that is ineligible for mixingwith another file, the ineligible portion being a part of an audiblesection of the audio file where overlaying additional audio is notpermitted; generate instructions for mixing audio data consistent withthe mixing information; and generate a stream of audio data based on thegenerated instructions.
 20. The device of claim 19, wherein thegenerated audio data corresponds to at least one of a file and a streamdelivered to a network.